As stated in Part 1, we use lua-TestMore for our testing and reporting. The output format is called TAP (Test Anything Protocol) . It is human readable and simple. TestMore and TAP are widely used enough that there are tools available to help you use it.
Using lua-TestMore for everybody (Corona and Lua users alike)
Using lua-TestMore is pretty easy. We only use a few functions. Here is a really simple example (which may be found main.lua in the BasicTest directory in our provided example code):
-- This is for our slightly modified scripts. -- Use require('Test.More') if you use the official lua-TestMore files. require('More') -- Specify the number of tests you plan to run. plan(2) local someValue = 1 local test_count = 0 -- This function verifies parameter 1 is equal to parameter 2 is(someValue, 1, "someValue should be equal to 1") test_count = test_count + 1 -- This function verifies parameter 1 is not equal to parameter 2 isnt(someValue, nil) test_count = test_count + 1 -- declare you are done testing and the number of tests you have run done_testing(test_count)
In a nutshell, you use plan() to declare how many tests you plan to run. When you are done testing, you call done_testing() with the number of tests you actually ran. (If the numbers don’t match lua-TestMore will be able to report that the test failed.) Then you can use the is() function which tests equality. As a convenience, there is also an isnt() function which tests inequality.
Here is the TAP output for the above test:
1..2 ok 1 - someValue should be equal to 1 ok 2
Just for comparison, let’s change our test so it always fails. Let’s delete the line:
local someValue = 1
The TAP output now looks like:
1..2 not ok 1 - someValue should be equal to 1 # Failed test (/Users/ewing/TEMP/LuaTestMoreCoronaExamples/BasicTest/main.lua at line 9) # got: nil # expected: 1 not ok 2 # Failed test (/Users/ewing/TEMP/LuaTestMoreCoronaExamples/BasicTest/main.lua at line 13) # got: nil # expected: anything else
There are also tools that help parse the output which help make automation even easier, (e.g. send emails when failed, display pass/fail results on a web page, etc.). We use Jenkins for our build and testing process and use plugins to help parse our output and report the results.
Using lua-TestMore with Corona:
lua-TestMore is written in just Lua with no external dependencies except the optional LuaSocket support. Because of this, you can use it directly in Corona. There is one caveat: lua-TestMore has implementation files in subdirectories and Corona currently doesn’t support Lua files in subdirectories. We have a modified version of lua-TestMore that flattens the directory structures which we provide with our example code on this page.
So like any Corona project, you just need a main.lua. From there, you will need a copy of our (flattened) lua-TestMore files. Just put them in the same directory as your main.lua. So the above example could be your Corona project. Just run it in the simulator, and it will run like any other Corona program. Try running the BasicTest project from our example code in Corona. You should see results as described earlier.
Output currently goes to stdout/stderr by default. You should launch with the Corona Terminal so you can see the output (until we finally fix this shortcoming.) (Internally, we launch from the command line, so we just need to look at stdout/stderr.)
Corona Specific Tests & Extra lua-TestMore support with Corona
So the first example script shown above is pretty generic. Corona users might like to see some real tests we use. So also included are two tests from our real automated test suite. Many of our tests have no display and are boring to watch and may end in less than a second, so I picked a test that displays to the screen and a test that plays audio. Our automated system is currently simple and unsophisticated so we don’t yet have the ability to do image comparisons of the display output or compare audio output to a recorded sample, but we can at least run tests on the states of things provided by our API functions. The display test can verify our APIs are returning expected values for the position APIs. The audio test is actually one of our fancier tests and tests things like opening files, loading different audio formats, ensures the audio callback system is working, as well as checking that the audio APIs return sensible values. (Incidentally, the audio test has inadvertently caught regression bugs on our general callback system on several occasions for us.)
One thing to be aware of: all our Corona tests do:
We wrote a Lua file called CoronaTest.lua (included with the example download) which hides some of the implementation details for deciding on when to write to a socket, file, or stdout/stderr (which I’ll discuss a bit more in the next part). Since we have multiple tests, we didn’t want to rewrite all the lua-TestMore setup code for each test, so we put all the logic in our CoronaTest.lua file.
Also, we end all our tests with a call to os.exit(). This helps with our automation process. On devices, we need to quit and return to the home screen so we can launch the next test; there is no user available to hit the home button. On our simulator, we actually have a command line way of launching the Mac simulator so we can run each test. We throw in an os.exit() at the end of our test scripts to force Corona to terminate so we can invoke the next test. Some of our long time users might remember the simulator used to terminate with calls to os.exit() which was annoyance for most people so we only stop the simulation instead of letting Corona terminate. We have a secret command line switch that allows us to restore the original termination behavior which is convenient for our automated testing.
Some of our users have found some of our other command line switches. None of these are officially supported and we really don’t want people to use them because they sometimes need to change. We have already broken a few of our users the last time we made a major change. But when we work on command line parameters (for those monitoring our Daily Build check in logs), automated testing is usually the reason behind it. However, we aren’t trying to actively break our users either. So if you do use any of these switches (you probably can see them in our videos or search our forums), you use at your own risk.
lua-TestMore With Sockets
When testing on devices, getting the test output becomes more difficult. To achieve our goal, we modified lua-TestMore to support writing to sockets. Working with the lua-TestMore author, LuaSocket support is now available in the official lua-TestMore repository.
Sockets even work when running on the simulator, but it is not as convenient because you need to run the output listener (which we’ll introduce shortly). So we didn’t have to modify all our test scripts to deal with this, we modified our CoronaTest.lua so it first looks for the existence of a special file containing IP address and port information. If so, it uses it, otherwise, it falls back to either writing to a file (using a similar existence check), or falls back to stdout/stderr.
So to get going with sockets, you need to set up both a server and configure your test program so it can connect to the server. You will also need LuaSocket installed on your system.
Corona comes built in with LuaSocket. However, our example server code provided below uses LuaSocket and must run on your computer directly, not Corona, so you will need to install LuaSocket if you want to use our example server. (Though in retrospect, if one wanted to be really clever, the Corona Simulator could be written to be the server.)
There are many ways to install LuaSocket. I’m only going to document LuaCocoa.
1) Download the latest LuaCocoa from http://bitbucket.org/ewing/luacocoa/downloads
2) Copy the LuaCocoa.framework to /Library/Frameworks
3) Download my prebuilt LuaSocket binaries from http://www.assembla.com/spaces/xcodescriptingbridge/documents. The file is called LuaSocketLuaCocoa.zip.
4) Unzip the file and it should create a directory called LuaCocoa (inside there should be two subdirectories: PlugIns and Scripts)
5) Move that LuaCocoa folder (from Step 4) to /Library/Application Support
LuaCocoa provides a command line tools to run Lua programs with. In the terminal, you can type:
to execute a script.
* Note: Only the tool luacocoa knows how to find modules in the Application Support directory. The standard lua tool which is also provided only looks in the stock package paths, e.g. /usr/local.
First you need a server to receive the data now that you intend to redirect your output through the network. The server can do anything you want and be written in anything you want. To get you started, we are providing you a copy of our little server with our example download (TapSocketListener.lua). We took sample code from LuaSocket and modified it to become a simple server for us. Basically, our server waits for a connection. Once it has a connection, it reads data until we get a message that looks like a test completion or a client disconnect. We write all the messages that come through the socket to a file so we can process/analyze the results later.
-- Example Usage: lua TapSocketLisener.lua "*" 12345 ../test-output/mytest.tap true -- param IPaddress -- param port -- param testoutputfile -- param echoOn local socket = require("socket") require('os') host = host or "*" port = port or 12345 destinationFile = "/tmp/LuaTestMore_results.log" echoOn = true if arg then host = arg or host port = arg or port destinationFile = arg or destinationFile if arg == "false" then echoOn = false else echoOn = true end end print("opening file", destinationFile) print("Tap Listener echo is", echoOn) file = assert(io.open(destinationFile, "w+")) print("Binding to host '" ..host.. "' and port " ..port.. "...") file:write("# TapSocketListener started on host " .. tostring(host) .. ":" .. tostring(port) .. " to file: " .. tostring(destinationFile) .. ", echo is: " .. tostring(echoOn) .. "n") s = assert(socket.bind(host, port)) i, p = s:getsockname() assert(i, p) print("Waiting connection from talker on " .. i .. ":" .. p .. "...") c = assert(s:accept()) file:write("# TapSocketListener received connection.n") print("Connected. Here is the stuff:") l, e = c:receive() while not e do if true == echoOn then print(l) end file:write(l) file:write("n") if "# CoronaTest timeout triggered" == l then print("Timeout notification received...terminating") print(e) c:close() file:close() os.exit(1) elseif "# CoronaTest completed all tests" == l then print("Completed notification received...terminating") print(e) c:close() file:close() os.exit(0) end l, e = c:receive() end print(e) file:write("# TapSocketListener was disconnected unexpectedlyn") file:close() os.exit(2)
To run the script, you do something like:
/Library/Frameworks/LuaCocoa.framework/Versions/Current/Tools/luacoca TapSocketLisener.lua “*” 12345 /tmp/mytest.tap true
Regular Lua works too if you have everything installed in the right places:
lua TapSocketLisener.lua “*” 12345 /tmp/mytest.tap true
The first parameter is the IP address you want to use. Passing “*” as the IP address to our server will use your machines IP address so you don’t have to look it up. The second parameter is the port you want to use. You will need to pick a port number that is available on your system. (Valid ports range from 1024-65535.) The third parameter is the name and location of the file you want to write output to. The forth parameter you should set to true if you want to see output echoed to stdout, or false if you do not.
To help find a free port number, we have another script (FindFreeSocketPort.lua) to help you do that too. The code is really short for this one:
require('socket') local my_socket = socket.bind(arg or "*", 0) local ipaddress, port = my_socket:getsockname() my_socket:close() print(port)
In our automated process, we run FindFreeSocketPort.lua and feed the result into TapSocketListener.lua via shell scripts. You could combine the two programs into one if you find it more convenient but remember that your client needs to know the port number.
Client (Device) Side:
For generic lua-TestMore users (and not Corona users), you need to do the following:
1) In your script, use socket.connect to connect to the host and port of your server.
local server = socket.connect(host, port)
2) Require and initialize lua-TestMore’s socket support, passing in the server object returned by socket.connect:
For Corona Users, LuaSocket is already is built-in to Corona so you don’t need to install anything.
1) Remember to
2) Create a Lua file called TestMoreOutputServerInfo.lua and put it in your project directory. The file should look like the following:
module('TestMoreOutputServerInfo', package.seeall) Host = '192.168.192.125' Port = '60266' TimeOut = '300000'
You should replace the variables Host and Port with the correct values of where your server is.
TimeOut is our special self-destruct timer in Corona that we described in Part 1 to deal with runaway tests. This will call os.exit() after the number of milliseconds specified here expires. You should pick a number that is no shorter than the time it takes to complete your test.
That’s it. lua-TestMore should now automatically redirect the test output to the socket when you start running your test application. Remember to start your server before launching the app on your device.
These are the on-device videos from Part 1. If you look carefully at the terminal output, you can see the TAP output and the server scripts discussed in this article. The first test on both devices is the display object test which is included in the example download.
- We currently use symlinks or shell scripts to manually copy all our common lua-TestMore scripts into our test projects so we don’t need to maintain separate copies of the files. But this solution is one of the obstacles that is currently blocking us from incorporating these tests on Windows.
- Our self-destruct timer is actually implemented using the standard Corona timer.performWithDelay() API function.
We have shared this information for multiple reasons:
- Automated testing should be encouraged to promote higher quality software that we can all use
- To grow the community around Test More and lua-TestMore
- Fulfill a promise I made to document the LuaSocket support in lua-TestMore
- To enable Corona users to add their own tests for their own projects
- To improve our own software
To speak to the last points, our own tests are far from comprehensive. We have spent a lot of time getting a lot of other functionality enabled (such as automating Xcode), so there are plenty of coverage holes in our test coverage that we still need to fill. We want to continue to improve the quality of Corona and are working on many different fronts to achieve that. So as a new idea we want to pitch to our users, we’re suggesting that if you find a bug you want fixed more quickly, submitting an automated unit test that isolates the problem would be a great way to help us hit the ground running in fixing your bug.
There is a saying, every bug is really two bugs: the first is a bug in functionality; the second is a bug in the testing system because it should have been caught. So if once we confirm and fix your bug, we would add it to our automated system to prevent future possible regression problems.
Finally, if there is interest, we might be willing to make our Corona automated tests public. This will let you see what we are testing and not testing and let you perform our tests on devices we might not have. And if we share our repository under Git or Mercurial, we can take contributions from anybody to help improve our tests and cover bugs you want fixed.
At the very least, we hope some of our users find this information valuable in writing their own internal tests.
Please leave us feedback if there are things you would like or ideas you have.
Please stay tuned for Parts 4 & 5 where we focus on the Xcode iOS Simulator and Android.