... .  .    .       .joeberkovitz.com

Live (or not so live) on The Flex Show

Starting today, you can catch me on episode 63 of The Flex Show, in which the effervescent John Wilker and Jeffry Houser interview yours truly about Noteflight, Flex, and I forget what else — it was taped a few weeks ago, which feels more like several years given the wealth of activity that was packed into it. I think I might take a listen right now, to find out the exact manner in which I embarrassed myself.

1 Comment

A Flash “Clipboard” using Local Shared Objects

I spent a good part of yesterday implementing a new global clipboard feature for Noteflight using local shared objects or LSOs.  These cookie-like constructs are often used to persist application-specific data across multiple sessions, so that when you close an application down and then restart it later, it resumes in the same state you left it.  What I haven’t seen much of (and haven’t done before myself) is use LSOs to communicate data between completely different instances of the same application running at the same time.  It does work, and seems to work well.

Noteflight is a Flash-based music notation editor, and one of the most often-requested features has been the ability to cut, copy and paste between different documents.  These “clipboard” operations already worked within a single session editing a single document.  However, the clipboard contents did not survive across edits to a series of documents,  nor could they transfer between different documents open in different windows.  In effect, each copy of the application had its own private copy of the clipboard, initialized to emptiness.  Not surprisingly, users don’t like these limitations.

It seemed clear that LSOs would be a good vehicle for this enhancement.  We would write the clipboard to an LSO on every cut or copy, and read it out of the same LSO before every paste. This would not be exactly like a real native system clipboard (since it would be separate from the system clip that contains text, images, etc.) but for the purposes of Noteflight it would seem a lot like one. The challenges we expected to solve (and did) were as follows:  serialization, keeping LSOs small, handling LSO failure gracefully, and application version skew.  There were of course a couple of major surprises, too!

Serialization.  All LSOs are stored in files, and must serialize their data.  They do so using the AMF3 protocol. ActionScript 3 defines how one can manage the serialization of arbitrary Objects using the [RemoteClass] and [Transient] metadata tags (be sure to look those up if you are thinking of serializing objects in a Flex application).  I’ve used them before, but the special problem here was that the clipboard data structures had already been designed and were quite complex, with a number of potential references to objects sitting “out in the application”.  Serializing any of those would result in lots of irrelevant data getting “dragged in” to the LSO and bloating it, so they’d have to be marked as transient.  It would be also necessary to make sure that each and every class requiring a [RemoteClass] tag received it.  Getting all this right in a short amount of time seemed unlikely.

Fortunately I had plenty of existing code to serialize the objects in the clipboard to XML and back, since Noteflight persists musical scores as XML.  So I used Flash’s handy IExternalizable interface to take advantage of this. This is a great technique that allows you to decide how your objects will be serialized. Basically, what I did was kind of like this:

[RemoteClass]
public class AppClipboard implements IExternalizable
{
  private var _data:AppData;  // contents of the clipboard...

  public function writeExternal(output:IDataOutput):void
  {
    output.writeObject(XmlWriter.toXML(_data).toXMLString());
  }

  public function readExternal(input:IDataInput):void
  {
    _data = XMLReader.fromXml(new XML(input.readObject()))
  }
}

(Note the [RemoteClass] tag — you still need it even if you use IExternalizable, or Flash won’t reconsititute the class of your object when you read it back in again.)

With this code written, when you setting one of your shared object’s data properties to an AppClipboard, like this:

  mySO.data.contents = myAppClipboard;

then the writeExternal() method is implicitly used to serialize that object. Same deal goes when reading the shared object back in: readExternal() gets used.

Next challenge:

Keeping the data size low. since there is a default 100K limit on LSO storage. XML has its disadvantages, and verbosity is certainly one of them. I needed to compress my serialized objects somehow. I found a nice way to do that, which works for any serializable objects at all (whether you use IExternalizable or not). Here’s the “write side” of the technique:

  var so:SharedObject = getClipboardSO();
  var bytes:ByteArray = new ByteArray();
  bytes.writeObject(_clipboard);   // write our object to a ByteArray
  bytes.compress();    // compress the ByteArray
  so.data.contents = bytes;   // and write the ByteArray instead
  so.data.version = NotationXmlReader.CURRENT_VERSION;
  so.data.creationTime = _clipboardCreationTime;
  so.flush();

The “read side” works similarly:

  var so:SharedObject = getClipboardSO();
  if (so != null
      && so.data.version == CURRENT_VERSION
      && (_clipboard == null
          || _clipboardCreationTime < so.data.creationTime))
  {
      var bytes:ByteArray = so.data.contents as ByteArray;
      bytes.uncompress();
      _clipboard = bytes.readObject() as AppClipboard;
      _clipboardCreationTime = new Date().getTime();

      // Flush the object after reading it so it doesn't get cached!
      so.flush();
  }

Here you can see the version skew detection, as well as a guard against using a “stale” shared clipboard that is actually older than the local version. That can happen if the “write side” fails to save an LSO due to size limitations, in which case we have to degrade gracefully and use the local clipboard instead of the stale shared one. This means that a single instance will still behave consistently even if it fails to transfer data to other instances.

Surprise #2: note the call to so.flush() on the read side. I did not expect to have to flush() a SharedObject after reading it. However, before putting this flush in, I found that the player would cache the SharedObject and refuse to check to see if it had been updated when reloading it by calling SharedObject.getLocal(). This caused clipboard-style transfer of data between different application windows to fail sporadically. It took a long time to figure this one out.

No Comments

Flexcover 0.81 is out

A quick update: Alex Uhlmann and I have just released the latest Flexcover edition on our Google Code Project. This release supports the newly announced Flex 3.2 SDK, Flex Builder 3.0.2 and AIR 1.5.

No Comments

Vote for sampled audio to not crash IE7

Turns out there’s a nasty bug in sampled audio playback in the newly released Flash 10 player. Found by Tyler Wright of Legato Media, this bug takes down your entire IE7 browser if you open simultaneous windows with Flash content, one of which is playing back audio using the new SampleDataEvent API. This is a crippling problem for any ambitious Flash content on the web incorporating dynamic audio.

Fortunately, we now have the prospect of an American President who can speak in intelligible sentences… which means that you can turn all of your positive energy to getting Adobe to fix this bug. Please vote on it at:

https://bugs.adobe.com/jira/browse/FP-985

Future generations thank you! (Future generations of what, you may ask?)

1 Comment

Flexcover upgrade for Flex 3.2.0 SDK is looking good

Alex and I — lately, to be honest, it’s mostly Alex — have been working on the forthcoming upgrade release of Flexcover, the code coverage instrumentation tool for Flex/AS3. I’m feeling pleased because I finally merged up the modifications to the Flex SDK compiler to the 3.2.0 branch, in preparation for an expected release of a new Flex Builder in the MAX timeframe.

The merge went smoothly and the 3.2.0 compiler seems to work like a champ — well, at least, it does everything that the 3.0.x version did. What’s more, a sneak peek at Flex Builder 3.2 revealed that they’ve fixed an irksome bug in the Flex Compiler project options dialog, and it’s now possible to enable and disable coverage instrumentation using the compiler options field.

That’s just the part I did. Alex has been busy working on some very nice changes to the CoverageViewer, which should greatly reduce the performance problems people were seeing during data collection before. We’ve also made it much easier to merge reports from multiple test runs. A few more fixes are needed but things are looking up…

2 Comments

Strongly Typed Proxies in AS3

Something I’ve often wanted in AS3 is a strongly typed proxy: an object that implements a known interface with typed methods, but where every method results in the invocation of some method like invokeMethod(methodName:String, args:Array). AS3 does have a Proxy class, but unlike Java’s class of the same name, an AS3 Proxy doesn’t implement any particular interface. One can extend Proxy into a subclass that has an interface, but by that time the Proxy superclass isn’t delivering any real value. The AS3 Proxy has been designed for loose, dynamic typing — which after all is one of the strengths of the ActionScript/ECMAScript languages.

Why use a strongly typed proxy? A common use of such a proxy is to create a “smart wrapper” around some underlying object with a known interface that looks and acts (and code-hints) exactly like the underlying thingamabob, but does some other stuff around every method call or property access. You might want a smart wrapper that logs or records all the function calls (as well as executing them). You might want a wrapper that does a try/catch around every call. You might want a wrapper that serializes the call over a stream to some remote destination.

(If this sounds like what some call “aspect-oriented programming”, well, it is similar to one aspect of aspect oriented programming :-)

Anyway, we can’t use AS3’s Proxy to solve this problem, so what to do? We’ll start with a simple approach, but make it more general. If our interface looked like this:

public interface ISmashable
{
    function smash(force:Number):String;
}

and we wanted to create a smart wrapper that logged all function calls around any ISmashable implementation, we might create something like this:

public class SmashableLoggingProxy implements ISmashable
{
    private var _smashable:ISmashable;
    public function SmashableProxy(smashable:ISmashable)
    {
        _smashable = smashable;
    }

    public function smash(force:Number):String
    {
        trace("smash():", force);
        var result:String = _smashable.smash(force);
        trace("    result:",result);
        return result;
    }
}

Then, if we did something like var proxy:ISmashable = new SmashableProxy(realSmashable); we could treat the resulting proxy as an ISmashable everywhere in our application, Every time we called, say smash(5) method on proxy, not only would realSmashable.smash(5) be called but we’d see some nice trace output as well.
If you have the time to create a lot of wrapper methods like this (or write a fancy code generator to save the trouble), that’s fine.  Another way to go, however would be to create a “generic” smart wrapper base class: [Continued...]

9 Comments

Controlling Audio Latency in Flash 10

Adobe threw a slight curve ball with the long-awaited release of Flash Player 10 today.  The release itself wasn’t the curve ball (although not everyone expected it to happen on thisdate).  The surprise was the audio latency using the new dynamic sound generation APIs.

Audio latency, for programs that synthesize sound on the fly, is roughly defined as the amount of time between the program creating a sound and the computer’s audio output actually playing that same sound.  Latency is important because it affects the user experience.  If a user hits a “play” button and waits 1 second before hearing anything, that can be annoying.  If a user types a note into a music notation editor and has to wait 1 second before hearing anything, that’s downright maddening.

Latency isn’t all bad, though. The higher the latency, the more “give” the sound output pipeline has, and the more resistant it is to disruption by other processes running in your browser and on your computer. Ideally you want to strike a balance between latency and robustness in your sound output.

In Noteflight, the latency on Mac OS X had been at a comfortable 30 milliseconds — about a 30th of a second — using the Flash 10 beta.  But in the released production version, Noteflight’s audio latency suddenly jumped to about 850 milliseconds.  Other folks with audio synthesis apps reported other numbers, all of them different.  Whoa!  What happened?  (Before going further, let me say that our latency is now back at 250 milliseconds, so things aren’t as bad as they first seemed.)

Adobe introduced a new behavior into the released player, where the latency depends on the number of samples you provide to SampleDataEvent. If you provide 2048 samples per callback, your latency will be the minimum available on the platform (30 ms on Mac, ~250 on Windows). If you provide 8192 samples per callback, on the other hand, your latency will be in the neighborhood 1000 milliseconds. The amount of latency for in-between sample block sizes is a complex function of this number. Tinic Uro will hopefully publish more information about that function, which apparently involves a power-law nonlinear function of the block size combined with the native sound driver buffer size, but experimentally I have seen these numbers on Mac OS X:

block size latency (ms)
2048 30
3072 170
4096 250
8192 850

This is good stuff to know, and it’s something of a surprise to us all. I notched the sample size in Noteflight back down to 4096 in a hurry, and latency is acceptable once again. Or, semi-acceptable. I’d like to get it even lower, but I don’t want to do that in too much of a hurry without adequate testing and tuning of the synthesis pipeline.

[Important footnote (10/16/08):  You might think that by adjusting the sample block size dynamically, you can change the latency on the fly for a single Sound object that is streaming samples.  The answer is... sort of, but not in a useful way.  You can make the latency increase by making the sample block size larger, but if you make it smaller again, the latency will not go back down: it will stay at the higher value.  What is reported to work, but I haven't tried it yet, is to output sound using more than one Sound object, each with its own block size (and hence its own latency value).  The usefulness of that approach is also not clear.]

5 Comments

Noteflight: An Online Music Notation Editor

It’s my great pleasure to make an introduction: Noteflight, please say hello to the world. World, there’s something I’d like you to meet: Noteflight. I’m really pleased to be able to finally write this, because it’s been a long journey to get to this point!

Noteflight is a new kind of tool for musicians, composers and educators: an online music notation editor. With Noteflight, anyone can create, share and publish musical scores using nothing more than a web browser. This ease of editing, sharing and publishing are what make Noteflight so different from other notation editors: music notation can finally be used on the Web in as natural and flexible a way as text, images or videos.

Here’s an example of a score that was created in Noteflight, shared, and embedded in this post:

This isn’t just a picture of music, but an interactive music display; for example, you can select and listen to individual notes and measures. Click the logo to see the full-page score, which can be printed. To get a sense of how the score above was created using the Noteflight score editor, please watch this video:

Scores are saved on the Noteflight server, so you and anyone you share them with can access them from anywhere on the Web. The music can be heard as well as seen: Noteflight has its own integrated audio playback, that sounds consistent on every computer. And Noteflight keeps track of all the past versions of your scores, so you can always go back and see where you’ve been.

The Noteflight beta is free and we’re accepting signups. As with all beta software, Noteflight is constantly being improved and extended; this blog is the best place to find out what’s happening (and what’s going to happen next) with Noteflight. Besides filling out the score editing features of Noteflight, we are planning additional services and capabilities for sharing music and organizing content.

There will be more posts coming up here about the development and internals of Noteflight — I’ve had to wait on many of those because the product wasn’t officially launched yet! A few quick techie observations:

• The Noteflight score editor was built with Adobe Flex 3.0.1 and Flash CS3
• The Noteflight server side was built with Ruby on Rails 2, Apache and MySQL and is hosted on Amazon EC2 and S3.
• The Moccasin open-source editing framework (see previous post) is a distillation of many of the music-independent aspects of the Noteflight score editor.
• The StandingWave audio library, previously discussed here (but not yet open-source), is used by the music synthesizer internal to Noteflight.

13 Comments

Moccasin: a Flex framework for graphical editing apps

Lately I’ve been working on a set of projects (both my own and others) that are built on top of Moccasin, a new open source AS3 framework that I’ve developed for building graphical editors. Moccasin was distilled and refactored out of the first project that used these ideas so that I could use it elsewhere without IP issues, and it lives on Google Code. Moccasin still has a long way to go in terms of overview documentation and I have to ask a certain amount of forgiveness on that subject. I’m perhaps releasing it a bit prematurely, but I regard it as pretty useful in a practical sense since I’m now working on the third substantial application built on top of it. This post is an attempt to bite off a little bit of that overview problem and endow Moccasin with a little more value to other people.

If the current lack of great overview docs is the bad news, well, the good news about Moccasin is that it’s got a complete sample editor application called Simpleworld as part of the distribution, with both AIR and Flex versions. Both the Moccasin framework and the sample application source code are pretty well commented, also. Here’s the Simpleworld application itself, to give a concrete sense of what Moccasin’s all about — it’s an editor for documents that consist of a bunch of squares of different sizes (I don’t call this world “simple” for nothin’):

Here are some of the things you can do, which show off what Moccasin’s about… [Continued...]

8 Comments

Miscellany: Flash On Tap, Moccasin, and Flexcover news

This Sunday morning seems like a fine moment for one of my increasingly infrequent posts.  It’s a gentle cloudy morning in Philadelphia, where I’m about to attend the ISMIR 08 conference on music information retrieval.  Although I’ve lately been busier than I have ever been in my life, here in Philly I’m isolated enough from my accustomed surroundings that my recent work habits are not functioning.  That’s probably a good thing, for me.  About the product of those work habits I have a lot to say, but much of it must still remain discreetly veiled, as the phrase goes, for a very short while longer.  I do have some updates on some other topics and sideline projects though.

Flash on Tap has been rescheduled for the spring of ‘09.  This is some disappointing news to be sure, and I was really looking forward to both speaking and attending.  But the disappointment is definitely tempered by the realization that this is going to be a far more kick-ass event, now that it can happen at a better time.  It was a really tough decision for the organizers, and it’s tough on everyone who was involved in the conference but they absolutely did the right thing: regrouping was in everyone’s best interest.

I have a new open source project out on Google Code called Moccasin.  It deserves its own lengthy post, and I hope to get to that soon.  Moccasin is a Flex/AIR application framework aimed at a specific domain: visual object-oriented editing applications.  The term “object-oriented” here refers to what one sees in the UI — of course the code itself uses objects and classes.  Visio and PowerPoint are representative examples.  Moccasin uses an MVCS approach and builds in support for multilevel undo/redo, document marshalling and persistence, selection, direct mouse manipulation and visual feedback.  It also includes what I think is an interesting and unusual approach to model objects: models can be any value object classes with bindable properties, and need not extend or refer to any classes or interfaces in the Moccasin framework. This project needs a good overview document, but there is a complete working sample application and the whole deal is reasonably well commented.  I have built two very large projects on Moccasin, one of which is Noteflight, the other of which is an AIR-based editor for creating levels in a new 3D game (not mine). I can at least claim that it has saved me time.

Finally, the poor stepchild, Flexcover. It’s getting a nice upgrade soon with some goodies: support for Flex 3.1 (thank you Daniel Rinehart!), a new UI from Alex Uhlmann for filtering classes and packages, and a fix to the modified SDK to correctly instrument top-level global functions outside of classes.

Oh well, time to go. More later…

7 Comments