StandingWave: Live Musical Audio Synthesis in AS3

December 4, 2007 on 7:16 pm | In Flex, Music, Programming | 15 Comments

I’ve been working on a funky new homebrew software project called StandingWave: a musical audio processing engine built entirely in Flash. My goal in doing this is to explore a world of online interactive music applications in which audio is not merely played back, but generated on the fly — performed, in fact — from an underlying representation of musical events. Such applications might range from a traditional music notation editor to game-like music composition environments to… knows?

Computer music performance is hardly something new, of course. But embedding the capability in Flash, at this point in the world, can make musical applications accessible on the web and amenable to community use in a way that’s never been possible before. Think about what applications like Buzzword and Google Spreadsheets are doing for traditional “productivity apps”.

I’ve started with the audio engine because it’s an interesting technical challenge, although I’m working on some of the other pieces concurrently. I’ve put up an initial crude example that demonstrates sample-based waveform synthesis. This toy application can play back single notes, a chromatic scale and a sample MIDI file at various transpositions, tempos and volumes, and all of this is accomplished by actually synthesizing digital audio signals on the fly, starting from a set of recorded guitar samples and applying gain envelopes, frequency shifting and mixing. Musically, it’s hardly exciting, but it’s a start on the capabilities needed to concretely deliver music in the Flash Player with no external add-ons, and without leaning on the crappy, highly variable MIDI playback delivered by the browser’s native OS.

There’s no waveform audio output API in Flash, so how is this done? Read on…

The basic idea is as follows: write a SWF file into a ByteArray in memory as raw bytecodes that encode an audio stream as an embedded sound asset. This byte array is exactly what would be produced if, in the Flash IDE, one imported a sound into the Library and exported it for runtime ActionScript usage. Then, use Loader.loadBytes() to load this in-memory SWF. Once loaded, use getDefinitionByName() on the loaded SWF to instantiate the exported asset as a Sound object, at which point it can be played directly via the Flash media API.

There’s more to StandingWave than that, of course.  Internally, there is a pipeline of objects representing music sources, audio sources, filters, mixers and performance objects.  WAV and MIDI formats need to be handled.  There are abstract notions of pitch, rhythm, tempo and temperament.  The audio is processed in small chunks to avoid tying up the main event loop, and there are some very non-obvious tricks required to start audio chunks at the correct time, so that they are played successively with reasonable time synchronization. (Consider that the Flash Player doesn’t ever guarantee that some chunk of code will be run at any exact point in time, whether you’re listening for Timer events or enterFrame.) Although playback is almost flawless on my Mac, I’m still having some issues on Windows at the beginning of a playback sequence. There is lots more work to do.

(I came up with the audio-output trick independently, but I’m not the first one to think of it. The Popforge open source project is one earlier realization, and there may be others. Popforge is geared more to audio streaming than to musical event processing, and I think I’ve improved on its approach by not attempting to maintain a continuous digital audio output stream, but instead overlapping audio chunks that always start cleanly with a new musical event. The same engine (or rough approach) appears to power the recently announced Splice Music site which exhibits some of the same characteristics as the kind of apps I’m interested in building.)

Anyway, more to come, but I thought I’d get something initial out there as food for thought, while I continue to make progress.

15 Comments

15 Comments »

RSS feed for comments on this post. TrackBack URI

  1. Joe, this is really cool! I have to admit that I was only dimly aware that there was no built in API for music synthesis in the Flash world, so it’s great to know that people are working on it. I’m jealous, though — I was just complaining to my officemate last week that I had always wanted to work on music software and never really gotten the chance. I am also envious that you can find time for midnight projects!

    Slip me the code and I’ll hack in some old-fashioned typewriter sounds into Buzzword… :-)

    Comment by David Coletta — December 5, 2007 #

  2. This sounds fascinating. I will be checking back to see how you go.

    Where do you want to take this, eventually? Are you aiming for an AS3 audio library, or to build a specific application?

    It would be great to have an app that is extensible and allows plugins to be built – something like Max/MSP or Reactor. (Of course, that’s way in the future…)

    Comment by Joshua Mostafa — January 3, 2008 #

  3. [...] Berkovitz is working on an audio processing engine in AS3. Looks like it’s in the very early stages, but it’s interesting … [...]

    Pingback by » as3 audio synthesis — January 3, 2008 #

  4. Would it be possible to use some of your work to maybe feed an MP3 stream to sound?

    Basically, avoid the flash streaming memory leak by doing the streaming manually and pumping the bytecode into the audio stream?

    I’m guessing at the end of the chain we’re down to PCM and MP3 would have to be decoded by its own class?

    Comment by Jai — January 8, 2008 #

  5. I guess it would be possible, but MP3 would have to be decoded in non-native AS3 code which I suspect would probably be unacceptable in terms of performance. You are right, all the internal audio in StandingWave is PCM — it’s the only way to go for raw speed.

    Comment by joe — January 8, 2008 #

  6. This is awesome Joe. I’ve done a bit of work on tempo detection in AS3 and was very dismayed with the lack of support for waveform manipulation and especially synthesis. I’d love to have a chance to play with StandingWave – please release it!

    Comment by William — January 10, 2008 #

  7. I never came across this post before I read your comment at createdigitalmusic.com
    It is good to see, that other people are also putting research on this subject even you have to take a lot of unusual efforts to it. If you are interested to collaborate with our team, please contact me.

    Comment by Andre Michelle — March 4, 2008 #

  8. [...] of dynamically-generated sound, but also some experiments that show how it can still be done). This framework also looks promising and has a nice sample-based example to show what the guy wants to do. [...]

    Pingback by Fnk » Developer diaries » An audio synthesis primer — March 16, 2008 #

  9. Hi Joe!

    Any plans for releasing the code of StandingWave?

    Regards / Jonas

    Comment by Jonas Nyström — August 21, 2008 #

  10. I have a number of other higher-priority projects going on, but I hope to release StandingWave by the end of the year.

    Comment by joe — August 21, 2008 #

  11. Hi Joe,

    Just wondering how your’e getting on with your ’standing wave’ project.

    Comment by Tim — March 21, 2009 #

  12. Hi Joe,

    Do you have any tip on loop audio in Flash?

    Comment by David — May 8, 2009 #

  13. Looping audio is a matter of, well, generating the same sequence of samples over and over again. The loop might not have the same length as the sample window that you have to supply when the Flash Player requests more audio data, so there’s a slight bit of math involved, but not much.

    Comment by joe — May 8, 2009 #

  14. Hi There.

    I been trying out StandingWave and it is fantastic.

    I am using flash to compile and seems to work fine.

    I am having a few problems getting my head around the looping of audio files. Do you have any examples of how to play a looping audio file?

    cheers

    Comment by Sean — March 12, 2010 #

  15. Hi Joe,

    What a wonderfull library, it helped me a lot with something I am making. I’m making a drum sequencer like FL Studio and this lib fixed the timing issue! Great!
    But I have a question, how do I loop a ListPerformance? I’ve created the ListPerformance by iterating through a set of data and determine the starttimes etc and it works perfectly, but it only plays once. I’d like to see it loop, how do I do that?

    Thanks in advance!

    Comment by Jeffrey — March 26, 2010 #

Leave a comment

XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Entries and comments feeds. Valid XHTML and CSS.
All content copyright (c) 2006-2007 Joseph Berkovitz. All Rights Reserved.