Posted on by

Introduction to Corona audio, ALmixer, and OpenAL

As many of you know, our new audio engine in Corona is powered by OpenAL. However, the API we provide in Corona does not look like OpenAL because we tried to make things easier to use. The API is actually inherited from ALmixer which is a wrapper library around OpenAL which we use in Corona. The API of ALmixer itself was influenced by another library called SDL_mixer. Part of the design of ALmixer was to make OpenAL easier to use while still being able to allow access to OpenAL features. Corona’s first goal of adopting ALmixer was to pass on the ease of use of using OpenAL to our users. But we are also interested in passing on the power of OpenAL’s features to our users as well. The secret audio APIs in Corona are our initial thoughts/groundwork in making this happen.

So thus far in Corona, we have hidden the details of how OpenAL works and we let users just think of it as a sound playing engine. But to really appreciate what OpenAL can do, how it works, and how the secret APIs apply to you, it is worth describing the high level design of OpenAL.

The Design of OpenAL in a Nutshell

OpenAL has three fundamental object types: buffers, sources, and the listener.

Buffers are essentially your audio data. We hide all those details from you. In Corona, all you ever see is loadSound and loadStream which hides lots of complex code from you.

Sources are things that emit sounds. You might think of them as your game objects, like a missile which has a rocket sound or an exploding ship that has a boom sound.

The listener is essentially where your head & ears are in the game. Do not confuse this with the Corona notion of a listener which is a callback function. In OpenAL, the listener is literally the object that is listening to (hearing) all sounds in your application.

In Corona, thus far, you’ve been using the notion of “channels” to control different simultaneously playing sounds. But “channels” as you know them in Corona don’t really exist in OpenAL; we made them up. In OpenAL, each sound emitting object is a “source”. A source has multiple properties like gain (volume), position, velocity, etc. So the secret formula in Corona is that, every channel maps to a different OpenAL source object. And conversely, every OpenAL source maps to a Corona channel. Understanding this will let you exploit a backdoor API we exposed in Corona to OpenAL directly. This is where the fun begins.

Secret Audio APIs Part 1: luaal (Lua/OpenAL bindings)

We wrote a partial direct interface between Lua and OpenAL. This will let you call an assortment of OpenAL functions directly in Lua. There are several families of functions in OpenAL. The ones of interest to you are the functions that set properties on sources and the listener.

So for example, OpenAL has a function (in C) called alSourcef(ALuint source_id, ALenum param, ALfloat value); You might use this to set the pitch on a source, e.g.


alSourcef(source_id, AL_PITCH, 2.0);

The value 2.0 shifts the pitch up by one octave for the source you designate.

Another example is to set the position on a source with the alSource3f function:


alSource3f(source_id, AL_POSITION, 5.0, 0.0, 0.0);

This would set that source’s position to the coordinate <5,0,0> in your world.

Similarly, there are functions that control the listener which is where your head/ears are in the world. Similar to sources, the listener also has its own properties such as position, velocity, etc. Once you define properties on your sources and the listener, OpenAL automatically computes how the final audio output is supposed to sound and plays this output to your speakers.

alListener3f(AL_POSITION, 0.0, 0.0, 0.0);

This would set the listener (your ears in the game) position to <0,0,0> in your world (<0,0,0> also happens to be the OpenAL default). So if you play the above sound (source) at <5,0,0>, the sound will come out of the right speaker because the source is to the right of the listener.

With our Lua bindings, you could call the above like:


al.Source(source_id, al.PITCH, 2.0)
al.Source(source_id, al.POSITION,  5.0, 0.0, 0.0)
al.Listener(al.POSITION,  0.0, 0.0, 0.0)

See the bottom of this page for a list of more OpenAL functions and constants we exposed.

Secret Audio APIs Part 2: Mapping between Corona channels and OpenAL sources

But now the question is, ‘How do I get a source id’? Well, this is the other half of to the secret audio APIs in Corona. Well, as I stated earlier, every Corona “channel” maps to an OpenAL “source”. So you just need a way of converting a channel to a source. Well, we have two undocumented functions that do the mapping.

local mysource = audio.getSourceFromChannel(1)
local mychannel = audio.getChannelFromSource(mysource)

This is good, but we have even something better. Unbeknownst to everybody, we did something clever and took advantage of Lua’s multiple return arguments feature. So every public audio function in Corona that returns a channel, also secretly returns a source id as a second parameter.


local mychannel, mysource = audio.play( mySound )

Similarly, every public audio function in Corona that takes a ‘channel=}’ table parameter, also takes an alternative ‘source=’ table parameter:


audio.setVolume(0.5, {source= mysource})

With pride, I can say that this was all designed/planned in advance and not some horrible kludge. And all of this (including luaal) has been in Corona since the very first release of our new audio engine way back in Build 222!

OpenAL is stateful

One important ground rule to understand is that OpenAL is stateful, meaning, if you change a property, it stays that way until you change it again later. Some of you have noticed already in Corona that when you change a volume on a channel including using fade-in/fade-out, the volume is persistent until you change it again. This is the reason. So remember that when you change a property in OpenAL, it stays that way until you change it again.

The Long Term Goal

We want to expose OpenAL features to Corona, but we want to find the correct balance of ease of use and power. However, as some of you know, Google has completely dropped the ball on audio and most of our engineering resources have been diverted to improving the basics on Android instead of working on new features. So in revealing these secret APIs, we are interested in seeing what you come up with and how you use them. Intelligent feedback on the forums is welcome, but be aware we may be slow to respond if at all. And who knows, maybe, one of you will write something so good that we will ask to incorporate directly in Corona.

Final thoughts:

Please use with care. Again, none of this stuff is supported or even necessarily tested. We do reserve the right to change the APIs or remove them entirely, so don’t complain if we break you. Please don’t make us regret discussing these secret APIs.

Additional Information:

The OpenAL Programmers Guide: http://connect.creativelabs.com/openal/Documentation/OpenAL_Programmers_Guide.pdf – This is the official online guide for OpenAL.

Beginning iPhones Game Development: http://playcontrol.net/iphonegamebook - I wrote the world’s first and only book that attempts to cover OpenAL comprehensively. Chapter 11 covers all the OpenAL 3D features in detail.

ALmixer: http://playcontrol.net/opensource/ALmixer/ - open source

luaal - http://www.assembla.com/code/almixer_isolated/mercurial/nodes/Isolated/luaal/luaal.c?rev=c07dbd386ded62c8c832585c2c1d2e74eb44c34c – open source

ALexplorer: http://developer.anscamobile.com/code/alexplorer - A Corona example program from our very own Alix, showing off some of these features

Appendix A: Examples Summary

-- We secretly returned two values for functions that return channels
local mychannel, mysource = audio.play(mySound)

-- 1.0 is normal, 2.0 is up one octave, 0.5 is down one octave
al.Source(mysource, al.PITCH, 2.0);

-- moves the source position to left-of-center (x-axis) by 5 units
-- number values are x-axis, y-axis, z-axis
al.Source(mysource, al.POSITION,  -5.0, 0.0, 0.0)

-- Note that all the audio APIs support "source" as a key name where we've documented "channel" as a key name, e.g.
audio.setVolume(0.5, {source=mysource})

-- Also, we have two functions that get convert one to the other
mysource = audio.getSourceFromChannel(1)
mychannel = audio.getChannelFromSource(mysource)

Appendix B: Non-comprehensive list of OpenAL functions and constants we exposed (via luaal)

Here are some other functions we exposed:

al.Enable
al.Disable
al.IsEnabled
al.Get
al.GetError
al.GetEnumValue
al.Listener
al.GetListener
al.Source
al.GetSource
al.DopplerFactor
al.DopplerVelocity
al.SpeedOfSound
al.DistanceModel

Here is the mapping of some of the constants we exposed. Lua is on the left, C is on the right. In Corona, you need to prefix ‘al.’ to everything, e.g. PITCH is al.PITCH.

{ "NONE", AL_NONE },
{ "FALSE", AL_FALSE },
{ "TRUE", AL_TRUE },
{ "SOURCE_RELATIVE", AL_SOURCE_RELATIVE },
{ "CONE_INNER_ANGLE", AL_CONE_INNER_ANGLE },
{ "CONE_OUTER_ANGLE", AL_CONE_OUTER_ANGLE },
{ "PITCH", AL_PITCH },
{ "POSITION", AL_POSITION },
{ "DIRECTION", AL_DIRECTION },
{ "VELOCITY", AL_VELOCITY },
{ "ORIENTATION", AL_ORIENTATION },

{ "REFERENCE_DISTANCE", AL_REFERENCE_DISTANCE },
{ "ROLLOFF_FACTOR", AL_ROLLOFF_FACTOR },
{ "CONE_OUTER_GAIN", AL_CONE_OUTER_GAIN },
{ "MAX_DISTANCE", AL_MAX_DISTANCE },

{ "DOPPLER_FACTOR", AL_DOPPLER_FACTOR },
{ "DOPPLER_VELOCITY", AL_DOPPLER_VELOCITY },
{ "SPEED_OF_SOUND", AL_SPEED_OF_SOUND },
{ "DISTANCE_MODEL", AL_DISTANCE_MODEL },
{ "INVERSE_DISTANCE", AL_INVERSE_DISTANCE },
{ "INVERSE_DISTANCE_CLAMPED", AL_INVERSE_DISTANCE_CLAMPED },
{ "LINEAR_DISTANCE", AL_LINEAR_DISTANCE },
{ "LINEAR_DISTANCE_CLAMPED", AL_LINEAR_DISTANCE_CLAMPED },
{ "EXPONENT_DISTANCE", AL_EXPONENT_DISTANCE },
{ "EXPONENT_DISTANCE_CLAMPED", AL_EXPONENT_DISTANCE_CLAMPED },

For a comprehensive list and understanding of what’s exposed, you can look at the source code of luaal.c.


Posted by . Thanks for reading...

46 Responses to “The secret/undocumented audio APIs in Corona SDK”

  1. Chris

    Very nice!
    So now I could actually make a panning effect and simulate helicopter flights ;)

    Reply
  2. Taka

    “Google has completely dropped the ball on audio” – what exactly do you mean by this? I was just informed a few days ago that Android is terrible with precise audio because of latency issues within the driver… I never knew this since I have been workin my currently project, which is a productivity app that does not use audio. Haven’t moved onto game-making – yet!
    Could you elaborate – on what is “known” about this? Thanks!

    Reply
  3. Taka

    So Gingerbread (2.3) NDK r5 has 45ms latency? That’s terrible – It seems a reasonable latency is ~10ms…? What’s the latest plan/update from Google on improving this?
    It’s funny – since just a few days ago, I had tried out about 4 or 5 virtual piano apps on my son’s Tegra2-based Android tablet and I was totally not impressed with the key response times compared to my mother-in-law’s iPad1… I guess this is why…
    It sounds like they’ve been concentrating on Graphics acceleration in the NDK – last I heard, Android does not even use GPU-acceleration for the UI – one of the reason why its UI is not as smooth as iOS. It’s such a bummer…

    Reply
  4. ewing

    As I have been told, Android’s audio problems are system-wide and there is no time table on when things will be fixed. Some problems are in the kernel itself so these are probably not issues that are easily fixed. In addition, because device manufacturers are free to do anything, it is also very hard to improve audio performance. Finally, their old audio API was terrible. Their new OpenSL ES based API seems to be a big improvement, but it is still a work in progress and doesn’t get around the fundamental latency problems. OpenSL ES may solve other problems we have with Android audio though and would like to move to it, but it requires us to move beyond/drop Android 2.2. But getting people and manufacturers to upgrade their Android version is another Google failing that is also no secret.

    Reply
  5. Paul Allan Harrington

    Thanks for sharing this.

    I’m hopeful that this info will allow the Corona community to dig into the audio API and provide “intelligent discussion and constructive criticism” :P as well as the testing that will advance the Corona SDK.

    -Cheers!

    Paul Allan Harrington
    Visuals Work: { visualswork.com }
    Consulting Group: { pahgroup.com }

    Reply
  6. chris martone

    OMG thank you so much Carlos and the rest of the Corona staff! I have been waiting for something like this for so long and as it happens im in the middle of creating a music app right now! I am going to start playing around with this ASAP. Again thank you, hope i can manage some cool uses. Maybe i can finally port my app I made with GameSalad(yuck!) to Corona!

    Reply
  7. Alberto

    Great post, and to give some feedback I love this approach. Provide a simple to use interface that developers can quickly get going with and then provide additional functionality via access to the lower level library for developers that want more control Win/Win!

    Reply
  8. chris martone

    Has anyone been able to get the al.VELOCITY to work? I would like to increase the speed at which a sound is played, is this the right feature to manipulate? If you have achieved what im trying to do please post an example. Thanks so much! And thanks again to all you Corona folk, my birthday came early this year!

    Reply
  9. ewing

    OpenAL velocity is not what you think. Velocity represents the speed of a sound source (object) moving through space. Think of a missile traveling across the screen. Velocity values work in conjunction with the Doppler Effect to pitch shift the sound effect.

    To change the playback speed of a source, you can change the al.PITCH which happens to change the speed at which a sound is played back at. But be aware that OpenAL does not support pitch corrected speed scaling.

    Reply
  10. chris martone

    thanks so much for the info, whats funny is I already implemented al.PITCH but failed to notice that it changed the playback speed. Much appreciated!

    Reply
  11. popo

    …and I was so hoping these would allow advanced envelope/pitch editing :(

    Still cool though…

    Reply
  12. Adrian Harris Crowne

    One of the things that makes Corona superior to almost any other mobile development environment is the access to up-to-the-minute changes. Publishing this secret API is in the same spirit, and I hope that it is a positive experience for Ansca. As a developer, it is great to be able to see where things are going and try out new ideas even if they are just experiments. Thanks for trusting us with the information!

    Reply
  13. Philipp Lenssen

    Just to add: Thanks for disclosing this stuff here. I am really looking forward to play around with more real-time audio creation features, especially for the use of creating music generation toys, or just more interesting dynamic music in a game.

    Reply
  14. Alberto

    It’s not possible an echo effect with this API, isn’t it?
    It seems only adding EFX extension to openAL (with effect objects) could be, and i figure out this is not possible in Corona yet.

    Reply
  15. Eric Wing

    No EFX is currently not directly exposed. I am not opposed to the idea of taking patches to luaal. However, EFX may not be not available on all platforms so a patch needs to take this into consideration. For Android and Windows, we use OpenAL Soft. For iOS and Mac, we use Apple’s implementation which promote their own effects (ASA).

    Reply
  16. Alberto

    Thanks for your reply, Eric!
    Sorry to tell you, I don’t know what ASA is. Could you tell me, please?
    Is not possible an echo effect in a Corona app?

    Reply
  17. ewing

    ASA: Apple Spatial Audio Extension
    Note that ASA was only first introduced to iOS in 5.0.

    I don’t think dynamic echo is currently possible through the sound APIs (I don’t think we exposed enough paths through luaal yet.) If your echo requirements are static, you could simply bundle two versions of the sound, one normal and one with echo already applied.

    Reply
  18. John

    Are these undocumented apis working in the simulator in Windows or on Android devices? Using this code:

    local ch, src = audio.play(playbackSoundHandle, { onComplete=onCompleteSound })
    al.Listener( al.POSITION, xl, yl, zl )
    al.Source( src, al.POSITION, x, y, z )

    The sound always plays at the same volume on both channels (L and R), no matter how I set the position.

    Reply
    • ewing

      They should work, though it’s not horribly tested. But the code paths for this are identical on all platforms. Make sure your audio is mono.

      Reply
  19. John

    Thanks for your quick reply. I hadn’t considered making sure the audio files were mono. However, I’m still having no luck. I even explicitly set al.ROLLOFF_FACTOR to make sure it wasn’t set to 0 somehow. I also tried running ALExplorer with no luck (I had to install Corona Build 591 to get it to run). The sound would play, but equally through both channels no matter how I set the sliders. Here are my system details, if it might make a difference:

    Windows: Vista SP1
    Android: HTC Desire Android 2.2
    Corona: build 591 and 731

    Reply
  20. ewing

    Everybody: I also wanted to note that audio onComplete callbacks also pass back event.source as a convenience.

    Reply
  21. Alan

    I’m having the same problem that John had, in that no matter what values I set for the position of the sound and listener I get the same sound through both the left and right channels. I’ve tried stereo and mono sounds, but no luck.

    Has anyone ever actually got this to work in Corona? I had no problems changing the pitch of a sound, so thought the position would be just as simple.

    Reply
  22. Alan

    Turns out that using mono files DID work, the problem was the values I was setting to reposition the sound. I assumed it would use “game world” values, i.e. -screenWidth for left and +screenWidth for the right. Changing the values to -1 for left and +1 for right fixed it.

    Reply
  23. Alan

    Also, I can now ONLY have exactly 50/50 left/right, 100% left or 100% right. Even a tiny adjustment (0.0001) to the right makes the sound entirely shift to the right channel. This is still better than I had before, but I would’ve preferred an 80/20 split.

    Eric Wing: Thank you, I just downloaded the app and can see how changing the values affects the sound – but if I use the xyz values for my source and listener I don’t get quite the same effect. I’ve tried searching for the source code for OpenAL Explorer but have had no luck.

    Reply
  24. Greg

    Is there a good example available that plays a continuous sound effect of an engines increasing revs but that uses both:
    A) swapping between audio files and
    B) for minor changes on each audio file a pitch change

    In particular how to transparently handle item a) above. This is when too much use of pitch change makes it sound non realistic.

    Reply
  25. Eric Wing

    Greg: I recommend you move this to the forums.
    I suggest for two files, you might try using two separate channels (one for each sound) and pause/resume each as necessary. You can also manipulate each channel volume as necessary to maybe help blending the two.

    Reply
  26. Don

    I tried to study the SuperAudio class, but have not yet made sense of it. I don’t need 3-d spacial effects, but just the ability to pan a sound, slowly, from left to right while it is playing. My higher math skills aren’t what they were in grade eleven. I’ve tried simply adjusting the listener X-axis value, to no avail. I can make a sound come out on the left, center or right, but not change it while it plays. Does anyone have a simple example of panning left to right on the fly that works on IOS 6? Thanks in advance

    Reply
  27. kumar ks

    Hi ,
    I am new to corona, I am developing an app , where i am using lot of mp3 files for narration purpose.
    I am here using audio.loadStream() and audio.play() along with audio.pause(). What actually happens is my app is running for 9 to 12 times properly, after that it just stuck and i have to re launch the app . Whats the problem here , am i using the api properly , or anything i have to modify . pls tell me as soon as possible .

    Reply
  28. Danijel

    Hi, thanks for sharing.

    We at Little Endian have developed some cool sound effect SDKs, just looking into possibilities of adding them to Corona. Not sure if game devs would like it in Corona? Any feedback most welcome…

    Reply
    • Terry

      Danijel:

      I would love to see more audio effect functionality. We’re building children’s apps which would definitely benefit from functionality akin to OpenAL EFX or iOS ASA. Tell us more about what you’re thinking. ^_^

      Reply

Leave a Reply

  • (Will Not Be Published)