This piece is dedicated to the memory of George Russell.

I first heard his music in 1974 or so, when I was 14 or 15 years old and living in London. I would occasionally hang out with my father’s friend and colleague Erik Edvardsen, an electronic engineer. Erik lived in Brixton (which meant I got to go on the then-new Victoria Line to visit him, kind of a treat) and he had an adventurous taste in music. Over time, he lent me a whole bunch of albums which I took home and listened to. The ones I liked a lot, I’d record on cassette. I still have a lot of those cassettes, although I don’t have a way to play them now.

Some of the music he lent me I found inaccessible at first, and George Russell’s music fell into this category. Something about it held my attention though, and kind of riveted me to my spot and forced me to listen. I think Erik had a few of the early 60s albums in there — Stratusphunk, Sextet at the Five Spot. Anyway, I kept listening, and I began to connect with it, to the point where I didn’t really want to hear anything else for a while. Then I started getting hold of his other recordings. George’s writing became then, and still is, some of my favorite music of all time. It has a startling freshness, a willingness to explore, an openness to tonal possibilities, that to me has rarely been equaled in jazz or in music of any genre. And for all of its modernity, George’s music is never dry or academic: it’s emotional, hard-swinging music that has a lot of feeling. Another thing: there is nothing extra or gratuitous in it. Everything that is there is there for a reason, and you can directly experience those reasons by listening.

A couple of years later, as I continued studying and playing music while attending high school in Lexington, Mass., I learned about his “Lydian Chromatic Concept of Tonal Organization” — a system of thinking about harmonic and tonal relationships that changed the thinking of a lot of well-known jazz musicians from Miles to Coltrane to… well, a whole lot of people. I knew that George Russell lived in Cambridge and somehow screwed up my nerve to look him up in the phone book and call him up. I remember that it was kind of a weird stilted conversation, because I didn’t really know what I wanted to say to my idol; I guess this makes me kind of a teenage George Russell stalker. I think I stammered out something about wanting to find his book on the Lydian Concept, and he got me off the phone quickly and gracefully by telling me I could find his book at the Busy Bee bookstore (if I remember the name right), which was on Hemenway St. near Berklee College of Music. I drove into Boston and bought it immediately.

The book was a revelation to me. It instantly unified and clarified of a whole current of confused and half-formed thoughts that had been running through my brain about scales, chords, and above all, tonality: the concept of music “being in a key”. A whole lot of junk dropped away and what seemed like a Rube Goldberg apparatus of chord symbols and modes turned into a much simpler way of looking at things.

There’s a lot that could be and has been said about George’s theory, and I’ll try to use my own language to hint at what he did that was so important to me and to others. In essence, George had a way of considering sets of tones (like notes in a melody, in a chord, or in a scale) as intervallic structures. He had rules for what one might call “orienting” these structures: determining what tones in a structure act as a harmonic foundation for the others, and how those other tones relate to the foundation. He also laid out some rules for how tonal structures related to each other in terms of a kind of “distance”. These rules had a way of reducing myriad musical ideas and variations into a much smaller number of essential forms that could be thought about much more easily, and could then be used to generate even more ideas. They corresponded in an obvious way to inner musical perceptions familiar to jazz improvisors. George’s concepts anticipated later work in tonal perception by theorists inside the academic community like Fred Lerdahl and Ray Jackendorff: work that has yet to catch up with George’s theories, if measured by its practical value to musicians. I think the fact that academics did not openly acknowledge or build on George’s work left a bad taste in his mouth, although it was hard to be sure.

Fast forward two more years: In 1979, I left Wesleyan after a dismal year of liberal arts education for the New England Conservatory of Music in Boston, to study piano and composition and make a musical life for myself. The fact that George taught there was no small part of this decision. I knew I’d be able to take his class (which I did) and that I’d have a chance to study composition privately with him (which I also did).

I found out some important things by studying music with George. I did learn a bunch more about his system of thinking, but perhaps the most surprising and hardest-to-accept lesson was that the things I loved the most about his music did not come from his theory, were not acknowledged by him in any explicit way, and could not be learned from studying with him. There is a kind of essential pride in his music, a matter-of-fact willingness to make a musical statement boldly, asserting itself according to its own musical logic, in fact, defining the very terms of that logic. This pride, this courage, the simultaneous respect for and departure from convention — they can be married with the materials of his theory, but they are not by demanded by that theory (and do not require its adoption). These qualities are there in his work and they were there in his life. George was fond of saying, “You play off the top of your knowledge.” In other words, knowledge determines your starting point for creation, but where you finally end up is the result of playing — and I mean, literally, engaging in unconscious, joyous play — with that knowledge.

Thank you, George Russell, for everything you did. Your thinking and your artistry had a huge impact on me. I will miss you.

Here is the melodic statement from George’s Stratusphunk, one of the first pieces of his that I remember hearing. It’s a simple and economical theme, moving around only a little bit. Nevertheless, in its economy, it generates an entire world of surprising implications when heard against the surrounding blues harmony.

You heard it from me: these folks are writing some of the most badass music-related code on the planet.

The Echo Nest are experts in “machine listening”: they have developed a set of algorithms that crunch through raw audio media and extract a set of distinctive musical features. These features roughly describe what is happening in the music at a hierarchy of time durations (beat, measure, section), and from the features they can compute a notion of similarity between different pieces of music. This similarity metric drives the recommendation aspect of their business.

Naturally enough Noteflight and The Echo Nest have some mutual interests, hence our visit. Audio media and music notation are both descriptions of music, so our companies both think a lot about how those descriptions are related. It’s a tough problem to go from either description to the other, and no algorithm can perform either task anywhere near as well as a human musician.

Anyway, while I was over there they showed off a very cool music hacking tool called Remix which you can grab from Google Code. It’s basically a Python library that takes an audio file, analyzes it using Echo Nest wizardry, and then returns a data structure describing the audio down to the beat level. You can then mess with these beat-length samples based on their descriptive data, and reassemble them in bizarre and unexpectedly musical ways.

As an example, they played me a version of “Here Comes The Sun”, in a strangely filtered version in which only beats in the same key as the opening intro had been retained. The result was a odd, drone-like modification of the song in which the intro itself was intact, but then unfolded into a sequence of snippets from the song that were completely familiar but from which all harmonic motion had been precisely excised.

I then heard a Hall and Oates song that had had beats 2 and 4 surgically removed from every measure. The result? A weird double-time version in which the song form progressed at twice the normal speed, the lyrics were mostly unintelligible but with many recognizable syllables, and the entire song’s length was chopped in half. The latter aspect could be viewed as an improvement on the original.

As a code-on-the-spot challenge, I asked if they could put together a version that sorted all the beats by amplitude, putting the softest ones first and the loudest ones last. 5 minutes of Python hacking later, we were listening to a bizarre, long crescendo of segments from the song, seamlessly reassembled into a whole. The beginning consisted mostly of quieter instrumental chords or the unaccented syllables of words, while the end was a kind of synopsis of all the climactic moments in the song with kick drum or vocal accents. The whole song turned into a single musical gesture, reassembled from its fragments into something completely different but still wholly familiar sounding.

After months of waiting for an opportunity to open up in my schedule, I’ve finally managed to create and package the StandingWave2 audio synthesis engine for Flash. It’s now up on Google Code at http://code.google.com/p/standingwave/. Phew… about time!

People have been asking for an open source audio library for a long time. Because the original StandingWave1 became an integral part of Noteflight, I could not simply give the code away. StandingWave2, on the other hand, is a clean subset that the Noteflight team is happy to make available to the Flash community under the MIT OS license.

The basic ideas behind StandingWave are sources, filters and performances. Sources and filters are simple, self-contained objects that can be hooked up to create a kind of audio-processing/sequencing pipeline, and then rendered by a “player” object that encapsulates the Flash Player 10 audio API. Performances allow source/filter combinations to be delivered in a continuous sequence, with extremely precise timing. Obviously this can be used to play music, but it can also be used for all kinds of dynamic sound creation. And it’s relatively easy to extend StandingWave to add your own kinds of sources, filters and performances once you read the code and get the idea.

All the DSP algorithms are in pure AS3. They would certainly be faster in PixelBender or Alchemy, but we’ve opted to keep the approach simple and flexible for now so that it’s easy for people to extend.

I will be talking much more about StandingWave at 360Flex and Flash on Tap, so hope you can make it to either of those conferences. I’ll be posting the slides here.

About its quality and performance: as useful as this library is (thousands of people use it on Noteflight), it could be so much better. It’s fast, but it should be faster; it has a basic repertoire of sources and filters, but should be richer. The need for improvement is one of the main reasons we’re open sourcing it: we very much want others to contribute.

Happy audio coding!

Important update: After a lot of amazing work by Max Lord, who rearranged SW2’s internal organs to create a new, bionically altered library, StandingWave 3 was released in June 2010. This update to StandingWave uses Adobe Alchemy to compile C audio processing code directly into the Flash Player for maximum speed, and is generally much faster and more flexible than Standing Wave 2.  It also includes a notion of modulators that allows for continuous variation of many filter parameters.  Like this idea? I do, which is why it’s now the production audio library in Noteflight.  Go to the StandingWave 3 repository on GitHub.

I’m showing a few cool demos of this technology at the conference, including a Moccasin-based visual editor for a “musical shapes” world in which shapes represent tones, and a set of data entry forms that progress smoothly through an accompanying musical form as the user navigates. It’s been loads of fun working on this stuff, which is to say it’s not anything like work at all.

I guess I have to go back to work now.

Gosh darn it.

I was thinking the other day about Noteflight, and the most frequently asked question of all — so frequent, it should probably go in our FAQ: “How can you make money if your notation software is free”?

I’ll answer that question somewhat indirectly, with an observation followed by another question.

There is a constant trend in the evolution of software. Our expectations of software value received per dollar spent are constantly being raised, whether we are aware of it or not, and online use has a lot to do with it. Part of that process is a shift in perspective that I’ll summarize this way: “Yesterday’s application is tomorrow’s component.”

Let’s look at a familiar example: word processing. Way back when, there was a program called Microsoft Word. Hey, there still is — and it still ain’t cheap! But I’m talking about Word 2000 right now, not MS Office 2008. Its main toolbar looked like this at the time:

And here’s a toolbar from one of those ubiquitous online “rich text editors” that you see in your browser all the time now, everywhere from blogs to email programs to content management systems:

So, back to the original proposition: do you pay anything for an editor like this, that you use in an application whose main purpose is to do something else, that’s larger-scale and more important to you?

Of course you don’t pay for that. You unthinkingly click the “B” button to make your text bold, never giving a thought to the fact that Microsoft used to charge a steep price for functionality like that, back in the day. As you do this, you are not thinking, “wow, I’m doing word processing!” You are using the editor to write your friend an e-mail, or to create some course content for your students, or to make up a document that you are storing online in Adobe Buzzword, Google Docs, etc.

This neatly sums up what Noteflight is all about. What you should pay for isn’t the raw ability to compose and edit music notation on a computer: it’s the software around the editor that matters. Music notation software is going to be free now.

Audio latency, for programs that synthesize sound on the fly, is roughly defined as the amount of time between the program creating a sound and the computer’s audio output actually playing that same sound.  Latency is important because it affects the user experience.  If a user hits a “play” button and waits 1 second before hearing anything, that can be annoying.  If a user types a note into a music notation editor and has to wait 1 second before hearing anything, that’s downright maddening.

Latency isn’t all bad, though. The higher the latency, the more “give” the sound output pipeline has, and the more resistant it is to disruption by other processes running in your browser and on your computer. Ideally you want to strike a balance between latency and robustness in your sound output.

In Noteflight, the latency on Mac OS X had been at a comfortable 30 milliseconds — about a 30th of a second — using the Flash 10 beta.  But in the released production version, Noteflight’s audio latency suddenly jumped to about 850 milliseconds.  Other folks with audio synthesis apps reported other numbers, all of them different.  Whoa!  What happened?  (Before going further, let me say that our latency is now back at 250 milliseconds, so things aren’t as bad as they first seemed.)

Adobe introduced a new behavior into the released player, where the latency depends on the number of samples you provide to SampleDataEvent. If you provide 2048 samples per callback, your latency will be the minimum available on the platform (30 ms on Mac, ~250 on Windows). If you provide 8192 samples per callback, on the other hand, your latency will be in the neighborhood 1000 milliseconds. The amount of latency for in-between sample block sizes is a complex function of this number. Tinic Uro will hopefully publish more information about that function, which apparently involves a power-law nonlinear function of the block size combined with the native sound driver buffer size, but experimentally I have seen these numbers on Mac OS X:

block size latency (ms)
2048 30
3072 170
4096 250
8192 850

This is good stuff to know, and it’s something of a surprise to us all. I notched the sample size in Noteflight back down to 4096 in a hurry, and latency is acceptable once again. Or, semi-acceptable. I’d like to get it even lower, but I don’t want to do that in too much of a hurry without adequate testing and tuning of the synthesis pipeline.

[Important footnote (10/16/08):  You might think that by adjusting the sample block size dynamically, you can change the latency on the fly for a single Sound object that is streaming samples.  The answer is... sort of, but not in a useful way.  You can make the latency increase by making the sample block size larger, but if you make it smaller again, the latency will not go back down: it will stay at the higher value.  What is reported to work, but I haven't tried it yet, is to output sound using more than one Sound object, each with its own block size (and hence its own latency value).  The usefulness of that approach is also not clear.]

Noteflight is a new kind of tool for musicians, composers and educators: an online music notation editor. With Noteflight, anyone can create, share and publish musical scores using nothing more than a web browser. This ease of editing, sharing and publishing are what make Noteflight so different from other notation editors: music notation can finally be used on the Web in as natural and flexible a way as text, images or videos.

Here’s an example of a score that was created in Noteflight, shared, and embedded in this post:

This isn’t just a picture of music, but an interactive music display; for example, you can select and listen to individual notes and measures. Click the logo to see the full-page score, which can be printed. To get a sense of how the score above was created using the Noteflight score editor, please watch this video:

Scores are saved on the Noteflight server, so you and anyone you share them with can access them from anywhere on the Web. The music can be heard as well as seen: Noteflight has its own integrated audio playback, that sounds consistent on every computer. And Noteflight keeps track of all the past versions of your scores, so you can always go back and see where you’ve been.

The Noteflight beta is free and we’re accepting signups. As with all beta software, Noteflight is constantly being improved and extended; this blog is the best place to find out what’s happening (and what’s going to happen next) with Noteflight. Besides filling out the score editing features of Noteflight, we are planning additional services and capabilities for sharing music and organizing content.

There will be more posts coming up here about the development and internals of Noteflight — I’ve had to wait on many of those because the product wasn’t officially launched yet! A few quick techie observations:

• The Noteflight score editor was built with Adobe Flex 3.0.1 and Flash CS3
• The Noteflight server side was built with Ruby on Rails 2, Apache and MySQL and is hosted on Amazon EC2 and S3.
• The Moccasin open-source editing framework (see previous post) is a distillation of many of the music-independent aspects of the Noteflight score editor.
• The StandingWave audio library, previously discussed here (but not yet open-source), is used by the music synthesizer internal to Noteflight.

In differential calculus, an inflection point, or point of inflection (or inflexion) is a point on a curve at which the curvature changes sign. The curve changes from being concave upwards (positive curvature) to concave downwards (negative curvature), or vice versa. If one imagines driving a vehicle along the curve, it is a point at which the steering-wheel is momentarily “straight”, being turned from left to right or vice versa.

My steering wheel is turning, for darn sure. I have recently left the wonderful company for which I have been working, Allurent, to start my own business. It’s an exciting change and one that I didn’t quite see coming until it had almost occurred by itself. This was a decision that felt like it happened to me, not a decision that I consciously made. I have an idea that I am absolutely passionate about, and must work on, as disruptive as the consequences may be. It’s as simple as that.

I will be talking much more about my company — called Noteflight — quite soon. Noteflight is an online platform for creating, sharing, viewing and hearing music. It’s a serious application, aimed at people who perform, compose, study, teach or collect musical content. Expect some very substantial posts from me in the next month or two.

I am also working with the crack Boston development shop Infrared5. To say these are smart folks is a huge understatement; go take a look at their site, their team and their achievements. Besides being smart, they’re also really nice folks, and I’m very pumped about the work we’ll do together, and are already doing now. Plus… they have some great dogs hanging around the office. I mean, how can you beat that?

Lastly, one more bit o’ news: I’ll be speaking at the Flash on Tap conference in Boston this year. Hope you can come!