Audio latency, for programs that synthesize sound on the fly, is roughly defined as the amount of time between the program creating a sound and the computer’s audio output actually playing that same sound.  Latency is important because it affects the user experience.  If a user hits a “play” button and waits 1 second before hearing anything, that can be annoying.  If a user types a note into a music notation editor and has to wait 1 second before hearing anything, that’s downright maddening.

Latency isn’t all bad, though. The higher the latency, the more “give” the sound output pipeline has, and the more resistant it is to disruption by other processes running in your browser and on your computer. Ideally you want to strike a balance between latency and robustness in your sound output.

In Noteflight, the latency on Mac OS X had been at a comfortable 30 milliseconds — about a 30th of a second — using the Flash 10 beta.  But in the released production version, Noteflight’s audio latency suddenly jumped to about 850 milliseconds.  Other folks with audio synthesis apps reported other numbers, all of them different.  Whoa!  What happened?  (Before going further, let me say that our latency is now back at 250 milliseconds, so things aren’t as bad as they first seemed.)

Adobe introduced a new behavior into the released player, where the latency depends on the number of samples you provide to SampleDataEvent. If you provide 2048 samples per callback, your latency will be the minimum available on the platform (30 ms on Mac, ~250 on Windows). If you provide 8192 samples per callback, on the other hand, your latency will be in the neighborhood 1000 milliseconds. The amount of latency for in-between sample block sizes is a complex function of this number. Tinic Uro will hopefully publish more information about that function, which apparently involves a power-law nonlinear function of the block size combined with the native sound driver buffer size, but experimentally I have seen these numbers on Mac OS X:

block size latency (ms)
2048 30
3072 170
4096 250
8192 850

This is good stuff to know, and it’s something of a surprise to us all. I notched the sample size in Noteflight back down to 4096 in a hurry, and latency is acceptable once again. Or, semi-acceptable. I’d like to get it even lower, but I don’t want to do that in too much of a hurry without adequate testing and tuning of the synthesis pipeline.

[Important footnote (10/16/08):  You might think that by adjusting the sample block size dynamically, you can change the latency on the fly for a single Sound object that is streaming samples.  The answer is... sort of, but not in a useful way.  You can make the latency increase by making the sample block size larger, but if you make it smaller again, the latency will not go back down: it will stay at the higher value.  What is reported to work, but I haven't tried it yet, is to output sound using more than one Sound object, each with its own block size (and hence its own latency value).  The usefulness of that approach is also not clear.]

Noteflight is a new kind of tool for musicians, composers and educators: an online music notation editor. With Noteflight, anyone can create, share and publish musical scores using nothing more than a web browser. This ease of editing, sharing and publishing are what make Noteflight so different from other notation editors: music notation can finally be used on the Web in as natural and flexible a way as text, images or videos.

Here’s an example of a score that was created in Noteflight, shared, and embedded in this post:

This isn’t just a picture of music, but an interactive music display; for example, you can select and listen to individual notes and measures. Click the logo to see the full-page score, which can be printed. To get a sense of how the score above was created using the Noteflight score editor, please watch this video:

Scores are saved on the Noteflight server, so you and anyone you share them with can access them from anywhere on the Web. The music can be heard as well as seen: Noteflight has its own integrated audio playback, that sounds consistent on every computer. And Noteflight keeps track of all the past versions of your scores, so you can always go back and see where you’ve been.

The Noteflight beta is free and we’re accepting signups. As with all beta software, Noteflight is constantly being improved and extended; this blog is the best place to find out what’s happening (and what’s going to happen next) with Noteflight. Besides filling out the score editing features of Noteflight, we are planning additional services and capabilities for sharing music and organizing content.

There will be more posts coming up here about the development and internals of Noteflight — I’ve had to wait on many of those because the product wasn’t officially launched yet! A few quick techie observations:

• The Noteflight score editor was built with Adobe Flex 3.0.1 and Flash CS3
• The Noteflight server side was built with Ruby on Rails 2, Apache and MySQL and is hosted on Amazon EC2 and S3.
• The Moccasin open-source editing framework (see previous post) is a distillation of many of the music-independent aspects of the Noteflight score editor.
• The StandingWave audio library, previously discussed here (but not yet open-source), is used by the music synthesizer internal to Noteflight.

In differential calculus, an inflection point, or point of inflection (or inflexion) is a point on a curve at which the curvature changes sign. The curve changes from being concave upwards (positive curvature) to concave downwards (negative curvature), or vice versa. If one imagines driving a vehicle along the curve, it is a point at which the steering-wheel is momentarily “straight”, being turned from left to right or vice versa.

My steering wheel is turning, for darn sure. I have recently left the wonderful company for which I have been working, Allurent, to start my own business. It’s an exciting change and one that I didn’t quite see coming until it had almost occurred by itself. This was a decision that felt like it happened to me, not a decision that I consciously made. I have an idea that I am absolutely passionate about, and must work on, as disruptive as the consequences may be. It’s as simple as that.

I will be talking much more about my company — called Noteflight — quite soon. Noteflight is an online platform for creating, sharing, viewing and hearing music. It’s a serious application, aimed at people who perform, compose, study, teach or collect musical content. Expect some very substantial posts from me in the next month or two.

I am also working with the crack Boston development shop Infrared5. To say these are smart folks is a huge understatement; go take a look at their site, their team and their achievements. Besides being smart, they’re also really nice folks, and I’m very pumped about the work we’ll do together, and are already doing now. Plus… they have some great dogs hanging around the office. I mean, how can you beat that?

Lastly, one more bit o’ news: I’ll be speaking at the Flash on Tap conference in Boston this year. Hope you can come!

To run this demo, get hold of the FP10 standalone player, then use it to open this URL:

http://www.joeberkovitz.com/projects/StandingWave/physicalSound/physicalSound.swf

(If you try this with FP9, you’ll get a runtime error dialog, or the app won’t open. And at some point soon the FP10 API will probably change, invalidating this app and obliging me to rebuild it, so in the near future YMMV.)

The sync between the sounds and the visuals is quite delicate and if it is off by even a little bit, the results look terrible. On my MacBook Pro, the auto-calculation of the audio latency is near perfect, but on Windows the player API’s Sound.position property seems to be off by about 150 msec, enough to ruin everything. You can play around with the latency in this example by moving the slider.

I didn’t post source because I am using a proprietary library, but the sound code in the example is not very interesting or complex actually. The trickiness in this app is all in the audio/visual synchronization. I am actually running two identical physical simulations, one of which runs ahead of the other, generating each collision sound exactly the right time interval before the other simulation displays the same collision. You can see both simulations by clicking the checkbox in the upper left of the example.

I used an open source physics engine called APE (Actionscript Physics Engine) available, at http://www.cove.org/ape/index.htm. I picked it because it was very easy to modify the collision detection mechanism to add audio hooks, but I haven’t done enough research to compare it to other engines for this purpose.

Here’s a screenshot (without sound of course!):

I think this will be a fun one — my computer will be making a lot of noise and playing some tunes. Some of the highlights include:

• The strange and scary things you have to do to synthesize audio in Flash 9
• The big audio improvements in Flash 10 (code and demos will be shown!)
• Architectural approaches to dealing with latency and synchronization
• Adding simulated sound to a physics engine
• The brave new world of Flash applications for creating music, not just playing it back

Hope you can make it!

Computer music performance is hardly something new, of course. But embedding the capability in Flash, at this point in the world, can make musical applications accessible on the web and amenable to community use in a way that’s never been possible before. Think about what applications like Buzzword and Google Spreadsheets are doing for traditional “productivity apps”.

I’ve started with the audio engine because it’s an interesting technical challenge, although I’m working on some of the other pieces concurrently. I’ve put up an initial crude example that demonstrates sample-based waveform synthesis. This toy application can play back single notes, a chromatic scale and a sample MIDI file at various transpositions, tempos and volumes, and all of this is accomplished by actually synthesizing digital audio signals on the fly, starting from a set of recorded guitar samples and applying gain envelopes, frequency shifting and mixing. Musically, it’s hardly exciting, but it’s a start on the capabilities needed to concretely deliver music in the Flash Player with no external add-ons, and without leaning on the crappy, highly variable MIDI playback delivered by the browser’s native OS.

There’s no waveform audio output API in Flash, so how is this done? Read on…

The basic idea is as follows: write a SWF file into a ByteArray in memory as raw bytecodes that encode an audio stream as an embedded sound asset. This byte array is exactly what would be produced if, in the Flash IDE, one imported a sound into the Library and exported it for runtime ActionScript usage. Then, use Loader.loadBytes() to load this in-memory SWF. Once loaded, use getDefinitionByName() on the loaded SWF to instantiate the exported asset as a Sound object, at which point it can be played directly via the Flash media API.

There’s more to StandingWave than that, of course.  Internally, there is a pipeline of objects representing music sources, audio sources, filters, mixers and performance objects.  WAV and MIDI formats need to be handled.  There are abstract notions of pitch, rhythm, tempo and temperament.  The audio is processed in small chunks to avoid tying up the main event loop, and there are some very non-obvious tricks required to start audio chunks at the correct time, so that they are played successively with reasonable time synchronization. (Consider that the Flash Player doesn’t ever guarantee that some chunk of code will be run at any exact point in time, whether you’re listening for Timer events or enterFrame.) Although playback is almost flawless on my Mac, I’m still having some issues on Windows at the beginning of a playback sequence. There is lots more work to do.

(I came up with the audio-output trick independently, but I’m not the first one to think of it. The Popforge open source project is one earlier realization, and there may be others. Popforge is geared more to audio streaming than to musical event processing, and I think I’ve improved on its approach by not attempting to maintain a continuous digital audio output stream, but instead overlapping audio chunks that always start cleanly with a new musical event. The same engine (or rough approach) appears to power the recently announced Splice Music site which exhibits some of the same characteristics as the kind of apps I’m interested in building.)

Anyway, more to come, but I thought I’d get something initial out there as food for thought, while I continue to make progress.