... .  .    .       .joeberkovitz.com

Visiting with The Echo Nest

Yesterday I paid a head-spinning visit to The Echo Nest, a small software company in nearby Somerville, at the invitation of their CTO Brian Whitman. You might not have heard of The Echo Nest, but their products power an increasing number of music recommendation engines in sites around the world.

You heard it from me: these folks are writing some of the most badass music-related code on the planet.

The Echo Nest are experts in “machine listening”: they have developed a set of algorithms that crunch through raw audio media and extract a set of distinctive musical features. These features roughly describe what is happening in the music at a hierarchy of time durations (beat, measure, section), and from the features they can compute a notion of similarity between different pieces of music. This similarity metric drives the recommendation aspect of their business.

Naturally enough Noteflight and The Echo Nest have some mutual interests, hence our visit. Audio media and music notation are both descriptions of music, so our companies both think a lot about how those descriptions are related. It’s a tough problem to go from either description to the other, and no algorithm can perform either task anywhere near as well as a human musician.

Anyway, while I was over there they showed off a very cool music hacking tool called Remix which you can grab from Google Code. It’s basically a Python library that takes an audio file, analyzes it using Echo Nest wizardry, and then returns a data structure describing the audio down to the beat level. You can then mess with these beat-length samples based on their descriptive data, and reassemble them in bizarre and unexpectedly musical ways.

As an example, they played me a version of “Here Comes The Sun”, in a strangely filtered version in which only beats in the same key as the opening intro had been retained. The result was a odd, drone-like modification of the song in which the intro itself was intact, but then unfolded into a sequence of snippets from the song that were completely familiar but from which all harmonic motion had been precisely excised.

I then heard a Hall and Oates song that had had beats 2 and 4 surgically removed from every measure. The result? A weird double-time version in which the song form progressed at twice the normal speed, the lyrics were mostly unintelligible but with many recognizable syllables, and the entire song’s length was chopped in half. The latter aspect could be viewed as an improvement on the original.

As a code-on-the-spot challenge, I asked if they could put together a version that sorted all the beats by amplitude, putting the softest ones first and the loudest ones last. 5 minutes of Python hacking later, we were listening to a bizarre, long crescendo of segments from the song, seamlessly reassembled into a whole. The beginning consisted mostly of quieter instrumental chords or the unaccented syllables of words, while the end was a kind of synopsis of all the climactic moments in the song with kick drum or vocal accents. The whole song turned into a single musical gesture, reassembled from its fragments into something completely different but still wholly familiar sounding.

1 Comment

StandingWave2: an open source AS3 audio library

After months of waiting for an opportunity to open up in my schedule, I’ve finally managed to create and package the StandingWave2 audio synthesis engine for Flash. It’s now up on Google Code at http://code.google.com/p/standingwave/. Phew… about time!

People have been asking for an open source audio library for a long time. Because the original StandingWave1 became an integral part of Noteflight, I could not simply give the code away. StandingWave2, on the other hand, is a clean subset that the Noteflight team is happy to make available to the Flash community under the MIT OS license.

The basic ideas behind StandingWave are sources, filters and performances. Sources and filters are simple, self-contained objects that can be hooked up to create a kind of audio-processing/sequencing pipeline, and then rendered by a “player” object that encapsulates the Flash Player 10 audio API. Performances allow source/filter combinations to be delivered in a continuous sequence, with extremely precise timing. Obviously this can be used to play music, but it can also be used for all kinds of dynamic sound creation. And it’s relatively easy to extend StandingWave to add your own kinds of sources, filters and performances once you read the code and get the idea.

All the DSP algorithms are in pure AS3. They would certainly be faster in PixelBender or Alchemy, but we’ve opted to keep the approach simple and flexible for now so that it’s easy for people to extend.

I will be talking much more about StandingWave at 360Flex and Flash on Tap, so hope you can make it to either of those conferences. I’ll be posting the slides here.

About its quality and performance: as useful as this library is (thousands of people use it on Noteflight), it could be so much better. It’s fast, but it should be faster; it has a basic repertoire of sources and filters, but should be richer. The need for improvement is one of the main reasons we’re open sourcing it: we very much want others to contribute.

Happy audio coding!

5 Comments

Panel on Startups and Cloud Computing

With my business hat planted on my head, I’ve been invited to appear at a local panel on building startups with cloud-based computing services:

http://www.vilnashul.com/calendar/event/the-state-of-startups-using-cloud-computing/

This should be interesting; I know a couple of the other companies represented on the panel, and I am sure some diverting and unpredictable discussion is in the works! Also I’ve never been to the Vilna Shul, a much-neglected historic synagogue in Boston.

1 Comment

360Flex: Talking About Audio

I’ve been working all weekend on next Monday’s presentation at 360 Flex: the focus is on audio synthesis in Flash. I think this will be a fun one: I’ll be unveiling an open source version of Noteflight’s StandingWave audio library, at long last. This should make it much easier for folks to create Flash or Flex apps that do real-time audio synthesis, since it provides a set of useful building blocks on top of the raw Flash Player 10 API. StandingWave has a bunch of useful concepts in it like audio sources, filters/transformations and sequenced “performances” of timed events, and I’m hopeful that others will find it useful. I’ll post again when it’s actually ready for consumption — that is to say, next week!

I’m showing a few cool demos of this technology at the conference, including a Moccasin-based visual editor for a “musical shapes” world in which shapes represent tones, and a set of data entry forms that progress smoothly through an accompanying musical form as the user navigates. It’s been loads of fun working on this stuff, which is to say it’s not anything like work at all.

I guess I have to go back to work now.

Gosh darn it.

1 Comment

FITC talk slides: Building Complex Visual Editors / Moccasin

Here are the slides for my FITC 2009 talk, which covered aspects of building visual editors and the open-source Moccasin project.

http://joeberkovitz.com/projects/FITC2009/BuildingVisualEditors.pdf

No Comments

Noteflight wins FITC 2009 “Technical Excellence” award

Last night Noteflight won the “Technical Excellence in Flash” award at Flash In The Can (FITC) 2009 conference. I have to say, this is kind of cool! (I’m not sure what to call this award — does it have a name? Is it a “Flashy”? A “Canny”? A “Fitzi”?)

Overall, I am really enjoying this conference purely as a conference. It’s a much broader mix of creative and technical people than I usually see at software-related shows, the organization is great, and the quality of the presentations has been overall very high. A lot of the best Flash coders in the business are here, as are some folks from the Adobe Flash Player team. It’s a great bunch of folks to hang out with and talk to. As one highlight, it was nice to meet Robert Penner, a well-known name in tweening-weenie circles and one of the few other people who has worked on music notation in Flash (he did a lot of work on the mediaRAIN music viewer).

Toronto is a pretty hip and fun city too, as I am finding out. Maybe more on that later — although I didn’t bring my camera, stupidly.

No Comments

Vote for Noteflight (and come to FITC 09/Toronto)

I’ll be speaking at FITC ‘09 in Toronto later this month, which I’m really excited about. I will be talking about building graphical editors on the Flash platform (using Noteflight as well as other non-music-related editors as examples) and showing how to do this on top of the open-source Moccasin framework.

Anyway, I just returned from biking in Arizona to find that Noteflight had been nominated as a Finalist in the FITC 09 award category of Technical Excellence. FITC has a popular-vote aspect of these awards in which people can vote for a set of “People’s Choice” winners, separate from the judging process.

I hereby shamelessly ask for your vote at:

http://awards.fitc.ca/pc/

If elected, I promise there will be a chicken in every pot, plus a quick, painless end to the recession through a new government-mandated sedation program.

No Comments

Oops, I’m Speaking Again (at Boston Flex UG)

I’ll be talking on Tuesday, March 10 (that’s in 5 days) at the Boston Flex User Group’s monthly meeting. The subject: building graphical editors!

I’ll be talking about the architectural adventures encountered in building Noteflight, the first full-featured online music notation editor. I’ll then talk about how some of the ideas used there have been distilled into the Moccasin open source framework in Flex. I expect that we’ll build a new feature into a sample graphical editor right on the spot, as a live coding example. Finally, I’ll show a little bit of RedLine, a new interactive site-building tool based on Moccasin that I created with others at Infrared5.

Hope to see you there!

1 Comment

Why Music Notation is Free Now

(This is a cross-post from blog.noteflight.com.)

I was thinking the other day about Noteflight, and the most frequently asked question of all — so frequent, it should probably go in our FAQ: “How can you make money if your notation software is free”?

I’ll answer that question somewhat indirectly, with an observation followed by another question.

There is a constant trend in the evolution of software. Our expectations of software value received per dollar spent are constantly being raised, whether we are aware of it or not, and online use has a lot to do with it. Part of that process is a shift in perspective that I’ll summarize this way: “Yesterday’s application is tomorrow’s component.”

Let’s look at a familiar example: word processing. Way back when, there was a program called Microsoft Word. Hey, there still is — and it still ain’t cheap! But I’m talking about Word 2000 right now, not MS Office 2008. Its main toolbar looked like this at the time:

And here’s a toolbar from one of those ubiquitous online “rich text editors” that you see in your browser all the time now, everywhere from blogs to email programs to content management systems:

So, back to the original proposition: do you pay anything for an editor like this, that you use in an application whose main purpose is to do something else, that’s larger-scale and more important to you?

Of course you don’t pay for that. You unthinkingly click the “B” button to make your text bold, never giving a thought to the fact that Microsoft used to charge a steep price for functionality like that, back in the day. As you do this, you are not thinking, “wow, I’m doing word processing!” You are using the editor to write your friend an e-mail, or to create some course content for your students, or to make up a document that you are storing online in Adobe Buzzword, Google Docs, etc.

This neatly sums up what Noteflight is all about. What you should pay for isn’t the raw ability to compose and edit music notation on a computer: it’s the software around the editor that matters. Music notation software is going to be free now.

2 Comments

Flash 10 Audio in IE7 Recovers Consciousness

Adobe released their long-awaited fix to the dreaded FP-985 crasher bug in today’s Flash Player 10 update release, a/k/a Nemo (see my previous post on the subject).  The new player seems not to have broken anything else audio-related either.  Thanks, Flash team!  (Can we developers get a debug player, please?)

As to the other problems plaguing Flash 10 audio on Windows, well, the jury is still out on whether those have improved with the new player.  We at Noteflight have been receiving a regular stream of support emails complaining about output not working on Windows.  We have typically been asking these folks to turn on a configuration setting that reverts the audio output approach to Flash 9-based APIs, which always makes their problem go away.  Now that the new player’s been released, we’re going to try a different approach and ask our users to upgrade to the new Flash Player.  If this works, we’ll be really happy that Adobe fixed some bugs affecting configurations found in the field, but not in our lab.  If this doesn’t work, well… brace yourselves for some more long, dissatisfied posts here.  I’m hoping for the best, though.

No Comments