Now that you’ve seen the overview of the whole system, I’m going to talk about on-device testing on iOS first because this has been where we have endured the most pain.
The fundamental problem is that there are no good automated ways to install and launch an app to an iOS device. We don’t want to jailbreak our devices because that can cause other problems (like breaking push notifications and in-app purchases). I would love to be corrected (please tell me!), but otherwise, there are no command line tools Apple provides to help with this. Those who are wondering, the command line tool xcodebuild will not install to a device, and instruments (for UIAutomation) will not install an application to a device either.
I looked at iTunes, Xcode Organizer (3.x, 4.0) and the iPhone Configuration Utility. I saw nothing useful in iPhone Configuration Utility. Both iTunes and Xcode Organizer have drag-and-drop ways of installing an application, but drag-and-drop is not Applescriptable according to my talks with engineers at WWDC.
Digging around the Scripting Bridge dictionaries of the above apps. I found that Xcode alone seemed to have APIs that could build, install, and launch an application. So Scripting Bridge became the solution I pursued.
At the time, Xcode 3 was the current stable release and it was not known when Xcode 4 would ship. Little did I know that Xcode 4 would completely break and cripple their scripting support. So everything I present here only works in Xcode 3. I will share what I know about trying to get this to work in Xcode 4. My hope is that by sharing this information, enough demand might be created to get Apple to prioritize this kind of support, or somebody clever might figure out a different solution.
Scripting Bridge and LuaCocoa in nutshell
For those who are not familiar with Scripting Bridge, it is a technology launched in Mac OS X 10.5 Leopard that allowed other dynamic languages beside Applescript to talk to an application’s scripting dictionary which allows you to programmatically control an application. Certainly for me, I find Applescript to be a language that is easy to read, but impossible to write.
Officially, Apple supports Objective-C, Python (via PyObjC), and Ruby (via RubyCocoa or MacRuby) as alternative Scripting Bridge languages. But any language that has been bridged to Objective-C can be used. So the first thing to do is pick a language.
For this kind of work, Obj-C is a not so great language because there is almost no documentation for Xcode’s scripting dictionary so you need to experiment by calling methods and seeing what you get back. (You also have to generate header files and refer to them in your compile process.) The whole compile/link/launch/debug/repeat cycle can be slow for this, so interpreted languages have an advantage here.
Since the Corona programming language is Lua, and I am also the author of LuaCocoa (a bridge like PyObjC/RubyCocoa for Lua), it made a lot of sense for us to write our Scripting Bridge/Xcode script in Lua via LuaCocoa.
For those not familiar with Lua, it is a very easy language to learn. It is so easy to learn, that the majority of the video game industry uses Lua (from Monkey Island, to World of Warcraft, to Angry Birds) because their artists and content creators can learn it very quickly, be productive in it, and crank out levels, characters, items, etc. So don’t be deterred or intimidated by the Lua code presented here.
The final trick is to know how one language maps to the other.
As a simple trivial example, In Objective-C, you might do:
NSURL* file_url = [NSURL fileURLWithPath:@"/tmp/SomeFile.mp4" isDirectory:NO];
AVPlayer* av_player = [AVPlayer playerWithURL:file_url];
local file_url = NSURL:fileURLWithPath_isDirectory_("/tmp/SomeFile.mp4", false)
local av_player = AVPlayer:playerWithURL_(file_url)
Basically, bracket notation turns into a function call, colons are changed to underscores, and named parameters are flattened into a single function name.
So you can play with this, I am providing a trivial Xcode example project and simplified Xcode Scripting Bridge script to control it. A Git repository is at GitHub at git://github.com/ewmailing/XcodeScriptingBridge.git. A Mercurial repository is at Assembla at http://hg.assembla.com/xcodescriptingbridge. There is a sample Xcode project called MySampleProject.xcodeproj and a script called ScriptingBridge_Xcode.lua.
You will also need LuaCocoa. Download LuaCocoa here. Copy the LuaCocoa.framework to your /Library/Frameworks to make things easier. (My Xcode scripting bridge script assumes it can find the LuaCocoa command line tool at /Library/Frameworks/LuaCocoa.framework/Versions/Current/Tools/luacocoa, so if you want to put LuaCocoa elsewhere, you need to change the script.)
Since you have the entire script, you can play with it in detail. It was all written via experimentation. The way I do it is I first generate an Objective-C header file of the scripting interface for the application (Xcode).
sdef /Devloper/Applications/Xcode.app | sdp -fh --basename Xcode
This should create a header in your current directory called Xcode.h.
I then scan the header file looking for methods that look useful. Then I work backwards to figure out which methods will generate the correct objects I need to invoke the method I want. I use a lot of print and NSLog statements in my scripts when I write this stuff.
So if you scan the header, ultimately, I want to call launch() on the active executable. But experimentally, I discovered launch() is broken, so I must use debug(). But to call either of these, Xcode needs an executable first, so Xcode must build it. Calling buildStaticAnalysis_transcript_using_() on our Xcode project is the key. But we must make sure to build our product with all the correct settings. The rest of the script deals with that.
I will not go over every line for brevity, but will give a general overview of the script:
This launches Xcode 3. (Xcode 4 was renamed com.apple.dt.Xcode which actually makes things easier so I can have both Xcode 3 and 4 installed simultaneously.)
local xcode_application = SBApplication:applicationWithBundleIdentifier_("com.apple.Xcode")
This will open the specified Xcode project.
local xcode_document = xcode_application:open_(url_to_open)
The next thing is that we may need to change the “Active SDK”. This controls whether you are building for device or simulator. For this solution, we generally want device. Simulator does work, but we have a different solution for the simulator because there are different ways to install and invoke the simulator. This frees us from the Xcode 3 requirement and we can just use xcodebuild.
The following block of code is how I set the Build Configuration type. You get “Debug” and “Release” Build Configurations by default, but sometimes you create your own and need to specify different ones. I discovered the hard way that I need to get a direct reference to an existing object that has the configuration type I want in order to set it. So I use xcode_project:buildConfigurationTypes() to get an array of all the different configurations and search for the one I want by doing a string compare with their name() method.
-- Get the list of build configuration types and then hunt for the one I want.
local array_of_build_configuration_types = xcode_project:buildConfigurationTypes()
local desired_build_configuration_type = nil
for i=1, #array_of_build_configuration_types do
-- Xcode is acting weird. It seems to overwrite entries and replace them,
-- but I don't think it is actually
-- replacing them. We need to pick the right element.
-- For us, Release is the 2nd position, not the first.
if tostring(array_of_build_configuration_types[i]:name()) == tostring(build_configuration_type) then
desired_build_configuration_type = array_of_build_configuration_types[i]
Then, I check the current (active) build configuration and change it if I need to. It is questionable whether I need to actually set this here because the buildStaticAnalysis_transcript_using_() function takes this as a parameter. But I think I ended up needing this because buildStaticAnalysis_transcript_using_() didn’t seem to have an effect. But it might be worth somebody’s effort to confirm this.
-- Find out what the current (active) configuration is set to and change it if necessary
local active_build_configuration_type = xcode_project:activeBuildConfigurationType()
if active_build_configuration_type ~= desired_build_configuration_type then
Next we set the Active Target. Some projects have multiple targets so you need to make sure you are building the right one. For demonstration purposes, the sample Xcode project I provided has two targets, “OpenGLES1” and “OpenGLES2”. For purposes of this example, I will specify OpenGLES2. Like with the build configuration, we need to get a reference to an object so we iterate through an array looking for the item we want.
local array_of_targets = xcode_project:targets()
local active_target = xcode_project:activeTarget()
for i=1, #array_of_targets do
local the_name = array_of_targets[i]:name()
if tostring(the_name) == "OpenGLES2" then
active_target = array_of_targets[i]
Now we need to set the executable Xcode will launch/debug. Again, I have an “OpenGLES1” and “OpenGLES2” app, and will specify the latter. In this case, I got away with specifying the string name. But maybe doing it like the Active Target would be more reliable.
Now we can build it.
ret_string = xcode_project:buildStaticAnalysis_transcript_using_(false, false, desired_build_configuration_type)
And finally, we can launch it.
local ret_string = active_executable:debug()
That’s Xcode scripting in a nutshell. To try it for yourself, you should be able to just run the provided script:
Xcode should build and launch the app. The app never quits until you hit the home key. We talk about self-terminating the app in other parts (which basically call exit()).
Video 1: The video shows running the ScriptingBridge_Xcode.lua script. The script was altered to invoke the iOS Simulator instead of building to a device for the convenience of recording the video.
Video 2: For convenience, this is the same video shown in Part 1 of automated tests being run on an iPad. This time you might be able to make out some of the script debug output echoed in the console. Also notice that we quit Xcode after every run. This helps reduce Xcode hanging problems we sometimes encounter.
Other caveats and problems:
Xcode launch() is completely broken.
debug() also has a problem that the function doesn’t return immediately (or sometimes at all) after the application ends. I allow the Applescript to automatically timeout and resume. In our real tests, I also have other fallbacks to kill the Xcode process if our test run is taking too long.
Changing the Xcode settings programmatically seems to mess up the GUI and sometimes you will see targets and configurations renamed and duplicated instead of switched to the correct values. I usually keep a pristine copy of the Xcode project in revision control and never check in the one set through Scripting Bridge.
Sometimes Xcode fails to build, or install, or launch completely at random. Quiting/killing Xcode periodically (e.g. after every test) seems to help reduce this from occuring.
I could not get device selection to work if you had multiple iOS devices plugged in. If I recall correctly, the API needed for this wasn’t working correctly.
Unfortunately, about the time I got this script working for us, Xcode 4 was released, and all this no longer works. We currently keep Xcode 3 and 4 on the system and use Xcode 3 exclusively for testing. But with Lion and iOS 5 on the horizon, our Xcode 3 days are numbered.
I tried to write an Xcode 4 replacement script. But it appears that the scripting interface is incomplete and all the legacy stuff is broken. The new interface has too many holes which prevent it from being usable. Xcode 4 changes. Xcode 4 changes the paradigm to use Workspaces and Schemes. But there is no way to get to an object to build and execute. All the old APIs are marked deprecated, but they are actually completely broken. (Please help us file bug reports with Apple to fix the scripting API. The more reports they get, the higher priority this gets.)
Included with the sample project is a script named ScriptingBridge_Xcode4_broken.lua. Perhaps somebody out there might be able to make it work.
While the scripting bridge dictionary still seems broken, the Xcode Organizer finally introduces some buttons on the bottom of the interface which means there is an alternative to drag-and-drop. (Perhaps this is a result of my bug reports and desperate pleas for help at WWDC.) Automating this *might* be possible, though I believe this will require scripting to System Events services which I believe will be painful. (See Part 4 for writing Scripting Bridge to System Events for the Xcode iOS Simulator.)
In addition, this still fails to address how to automatically launch an app on an attached device. The command line tool ‘instruments’ seems to imply it can be used to launch applications on devices, but the documentation for the tool is almost non-existent. I have not been able to figure out how to specify devices with the -w switch and am unclear about specifying documents (especially since I don’t necessarily want to attach an instrument.)
But for now, this might be our only workaround, so I will be investigating this in the future. Anybody more familiar with (or just plain daring) System Events and the command line tool instruments and would like to contribute/share, please contact me!
In Part 3, we are going to discuss lua-TestMore and redirecting test output on devices through sockets. There is a lot of information that all readers regardless of background may be able to find something of value. Corona users in particular will find scripts they can directly use themselves, but there is enough information covered that all readers regardless of background may be able to find something of value.
Xcode and iOS readers, please stay tuned for Part 4 where we will return to Scripting Bridge and LuaCocoa to drive automated testing on Apple’s iOS Simulator that comes with Xcode.