Noteflight is a Flash-based music notation editor, and one of the most often-requested features has been the ability to cut, copy and paste between different documents.  These “clipboard” operations already worked within a single session editing a single document.  However, the clipboard contents did not survive across edits to a series of documents,  nor could they transfer between different documents open in different windows.  In effect, each copy of the application had its own private copy of the clipboard, initialized to emptiness.  Not surprisingly, users don’t like these limitations.

It seemed clear that LSOs would be a good vehicle for this enhancement.  We would write the clipboard to an LSO on every cut or copy, and read it out of the same LSO before every paste. This would not be exactly like a real native system clipboard (since it would be separate from the system clip that contains text, images, etc.) but for the purposes of Noteflight it would seem a lot like one. The challenges we expected to solve (and did) were as follows:  serialization, keeping LSOs small, handling LSO failure gracefully, and application version skew.  There were of course a couple of major surprises, too!

Serialization.  All LSOs are stored in files, and must serialize their data.  They do so using the AMF3 protocol. ActionScript 3 defines how one can manage the serialization of arbitrary Objects using the [RemoteClass] and [Transient] metadata tags (be sure to look those up if you are thinking of serializing objects in a Flex application).  I’ve used them before, but the special problem here was that the clipboard data structures had already been designed and were quite complex, with a number of potential references to objects sitting “out in the application”.  Serializing any of those would result in lots of irrelevant data getting “dragged in” to the LSO and bloating it, so they’d have to be marked as transient.  It would be also necessary to make sure that each and every class requiring a [RemoteClass] tag received it.  Getting all this right in a short amount of time seemed unlikely.

Fortunately I had plenty of existing code to serialize the objects in the clipboard to XML and back, since Noteflight persists musical scores as XML.  So I used Flash’s handy IExternalizable interface to take advantage of this. This is a great technique that allows you to decide how your objects will be serialized. Basically, what I did was kind of like this:

[RemoteClass]
public class AppClipboard implements IExternalizable
{
  private var _data:AppData;  // contents of the clipboard...

  public function writeExternal(output:IDataOutput):void
  {
    output.writeObject(XmlWriter.toXML(_data).toXMLString());
  }

  public function readExternal(input:IDataInput):void
  {
    _data = XMLReader.fromXml(new XML(input.readObject()))
  }
}

(Note the [RemoteClass] tag — you still need it even if you use IExternalizable, or Flash won’t reconsititute the class of your object when you read it back in again.)

With this code written, when you setting one of your shared object’s data properties to an AppClipboard, like this:

  mySO.data.contents = myAppClipboard;

then the writeExternal() method is implicitly used to serialize that object. Same deal goes when reading the shared object back in: readExternal() gets used.

Next challenge:

Keeping the data size low. since there is a default 100K limit on LSO storage. XML has its disadvantages, and verbosity is certainly one of them. I needed to compress my serialized objects somehow. I found a nice way to do that, which works for any serializable objects at all (whether you use IExternalizable or not). Here’s the “write side” of the technique:

  var so:SharedObject = getClipboardSO();
  var bytes:ByteArray = new ByteArray();
  bytes.writeObject(_clipboard);   // write our object to a ByteArray
  bytes.compress();    // compress the ByteArray
  so.data.contents = bytes;   // and write the ByteArray instead
  so.data.version = NotationXmlReader.CURRENT_VERSION;
  so.data.creationTime = _clipboardCreationTime;
  so.flush();

The “read side” works similarly:

  var so:SharedObject = getClipboardSO();
  if (so != null
      && so.data.version == CURRENT_VERSION
      && (_clipboard == null
          || _clipboardCreationTime < so.data.creationTime))
  {
      var bytes:ByteArray = so.data.contents as ByteArray;
      bytes.uncompress();
      _clipboard = bytes.readObject() as AppClipboard;
      _clipboardCreationTime = new Date().getTime();

      // Flush the object after reading it so it doesn't get cached!
      so.flush();
  }

Here you can see the version skew detection, as well as a guard against using a “stale” shared clipboard that is actually older than the local version. That can happen if the “write side” fails to save an LSO due to size limitations, in which case we have to degrade gracefully and use the local clipboard instead of the stale shared one. This means that a single instance will still behave consistently even if it fails to transfer data to other instances.

Surprise #2: note the call to so.flush() on the read side. I did not expect to have to flush() a SharedObject after reading it. However, before putting this flush in, I found that the player would cache the SharedObject and refuse to check to see if it had been updated when reloading it by calling SharedObject.getLocal(). This caused clipboard-style transfer of data between different application windows to fail sporadically. It took a long time to figure this one out.

Apache2. Apache2 prefork is configured to keep a certain number of processes running to respond to HTTP requests that come in. These processes are reused across a number of requests and thus tend to grow over time in their memory consumption to reach the largest amount of RAM demanded by any page served. When using mod_php, the RAM consumption is related to PHP application resources, in this case WordPress. I found that my Apache processes generally were in the 20m range for resident memory size (the RSS column in the ‘top’ display). With the out-of-box Apache2 configuration allowing as many as 15-20 processes, that meant that any large batch of concurrent requests (such as those originating from a search bot) would totally nail my machine. So the most important change I wound up making was to reduce the number of processes:

<IfModule mpm_prefork_module>
    StartServers          3
    MinSpareServers       3
    MaxSpareServers       3
    ServerLimit          10
    MaxClients           10
    MaxRequestsPerChild   1000
</IfModule>

This generally causes me to see around 5-6 processes running, with the possibility of as many as 10.

A number of folks have suggested I run nginx or lighttpd, which I’m sure would be a great idea, but I was strapped for time and in a mood to stick with things that have lots of online recipes and that I already somewhat understood.

Rails. Various approaches are available here as well. At first I was running the mod_passenger module, but I found this gave me a few 30m processes as opposed to the single 50m process I could get by running a single mongrel. So now I’m running a single mongrel and it’s working fine.

MySQL. Haven’t tuned this much yet, but I’m not seeing any problems.

WordPress. Caching is enabled on the advice of a number of people that this will decrease MySQL resource consumption.

Fortunately, we now have the prospect of an American President who can speak in intelligible sentences… which means that you can turn all of your positive energy to getting Adobe to fix this bug. Please vote on it at:

https://bugs.adobe.com/jira/browse/FP-985

Future generations thank you! (Future generations of what, you may ask?)

The merge went smoothly and the 3.2.0 compiler seems to work like a champ — well, at least, it does everything that the 3.0.x version did. What’s more, a sneak peek at Flex Builder 3.2 revealed that they’ve fixed an irksome bug in the Flex Compiler project options dialog, and it’s now possible to enable and disable coverage instrumentation using the compiler options field.

That’s just the part I did. Alex has been busy working on some very nice changes to the CoverageViewer, which should greatly reduce the performance problems people were seeing during data collection before. We’ve also made it much easier to merge reports from multiple test runs. A few more fixes are needed but things are looking up…

Why use a strongly typed proxy? A common use of such a proxy is to create a “smart wrapper” around some underlying object with a known interface that looks and acts (and code-hints) exactly like the underlying thingamabob, but does some other stuff around every method call or property access. You might want a smart wrapper that logs or records all the function calls (as well as executing them). You might want a wrapper that does a try/catch around every call. You might want a wrapper that serializes the call over a stream to some remote destination.

(If this sounds like what some call “aspect-oriented programming”, well, it is similar to one aspect of aspect oriented programming :-)

Anyway, we can’t use AS3’s Proxy to solve this problem, so what to do? We’ll start with a simple approach, but make it more general. If our interface looked like this:

public interface ISmashable
{
    function smash(force:Number):String;
}

and we wanted to create a smart wrapper that logged all function calls around any ISmashable implementation, we might create something like this:

public class SmashableLoggingProxy implements ISmashable
{
    private var _smashable:ISmashable;
    public function SmashableProxy(smashable:ISmashable)
    {
        _smashable = smashable;
    }

    public function smash(force:Number):String
    {
        trace("smash():", force);
        var result:String = _smashable.smash(force);
        trace("    result:",result);
        return result;
    }
}

Then, if we did something like var proxy:ISmashable = new SmashableProxy(realSmashable); we could treat the resulting proxy as an ISmashable everywhere in our application, Every time we called, say smash(5) method on proxy, not only would realSmashable.smash(5) be called but we’d see some nice trace output as well.
If you have the time to create a lot of wrapper methods like this (or write a fancy code generator to save the trouble), that’s fine.  Another way to go, however would be to create a “generic” smart wrapper base class:

public class SmartProxy
{
    private var _obj:Object;
    private var _functionNames:Dictionary;

    public function SmartProxy(obj:Object)
    {
        _obj = obj;

        // set up Dictionary that goes from functions in this wrapper to function names
        _functionNames = new Dictionary();
        var typeInfo:XML = describeType(this);   // returns metadata for subclass interface
        for each (var functionInfo:XML in typeInfo.method)
        {
            var name:String = functionInfo.@name;
            _functionNames[this[name]] = name;
        }
    }

    protected function handleInvocation(args:Object):*
    {
        // 'callee' is a special AS3-defined property of an 'arguments' object
        // that provides the Function which was called
        var functionName:String = _functionNames[args.callee];
        return applyFunction(functionName, args);
    }

    protected function applyFunction(functionName:String, args:Object):*
    {
        // This looks up the name of the function that was called, uses it to
        // obtain the Function in the underlying object as a property, and calls it.
        return (_obj[functionName] as Function).apply(_smashable, args);
    }
}

Now extend this to implement our interface:

public class SmashableProxy extends SmartProxy implements ISmashable
{
    public function SmashableProxy(smashable:ISmashable)
    {
        super(smashable);  // sounds like a Nintendo game, eh?
    }
    public function smash(force:Number):String
    {
        return handleInvocation(arguments);  // arguments is a special AS3 language keyword!
    }

    // (if our interface had other functions, they would all have the same body!)
}

The above SmashableProxy acts exactly like the underlying object. What’s the point? Well, given a base class of this kind, one can create smart wrappers of any variety with very little additional effort:

public class SmashableLoggingProxy extends SmashableProxy
{
    public function SmashableProxy(smashable:ISmashable)
    {
        super(smashable);
    }

    override protected function applyFunction(functionName:String, args:Object):*
    {
        trace(functionName, "():", args);
        var result:String = super.applyFunction(functionName, args);
        trace("result:",result);
        return result;
    }
}

The astute reader may notice that logging is generic, and wonder if it should be handled in the SmartProxy, not in this subclass.  So a further extension of this idea might  introduce a separate invocation handler that can supply generic handling for all method invocations,  and is a property of the SmartProxy.  Java in fact has exactly such a class for its Proxies.

In another post soon, I’ll describe how this fits into the testing strategy for Noteflight, which has a large automated test suite of recorded UI actions. We use this suite to validate the program before every checkin — it is a very complex editor, and it’s all too easy to fix one thing and break another, so the test suite makes sure we don’t do that!

Audio latency, for programs that synthesize sound on the fly, is roughly defined as the amount of time between the program creating a sound and the computer’s audio output actually playing that same sound.  Latency is important because it affects the user experience.  If a user hits a “play” button and waits 1 second before hearing anything, that can be annoying.  If a user types a note into a music notation editor and has to wait 1 second before hearing anything, that’s downright maddening.

Latency isn’t all bad, though. The higher the latency, the more “give” the sound output pipeline has, and the more resistant it is to disruption by other processes running in your browser and on your computer. Ideally you want to strike a balance between latency and robustness in your sound output.

In Noteflight, the latency on Mac OS X had been at a comfortable 30 milliseconds — about a 30th of a second — using the Flash 10 beta.  But in the released production version, Noteflight’s audio latency suddenly jumped to about 850 milliseconds.  Other folks with audio synthesis apps reported other numbers, all of them different.  Whoa!  What happened?  (Before going further, let me say that our latency is now back at 250 milliseconds, so things aren’t as bad as they first seemed.)

Adobe introduced a new behavior into the released player, where the latency depends on the number of samples you provide to SampleDataEvent. If you provide 2048 samples per callback, your latency will be the minimum available on the platform (30 ms on Mac, ~250 on Windows). If you provide 8192 samples per callback, on the other hand, your latency will be in the neighborhood 1000 milliseconds. The amount of latency for in-between sample block sizes is a complex function of this number. Tinic Uro will hopefully publish more information about that function, which apparently involves a power-law nonlinear function of the block size combined with the native sound driver buffer size, but experimentally I have seen these numbers on Mac OS X:

block size latency (ms)
2048 30
3072 170
4096 250
8192 850

This is good stuff to know, and it’s something of a surprise to us all. I notched the sample size in Noteflight back down to 4096 in a hurry, and latency is acceptable once again. Or, semi-acceptable. I’d like to get it even lower, but I don’t want to do that in too much of a hurry without adequate testing and tuning of the synthesis pipeline.

[Important footnote (10/16/08):  You might think that by adjusting the sample block size dynamically, you can change the latency on the fly for a single Sound object that is streaming samples.  The answer is... sort of, but not in a useful way.  You can make the latency increase by making the sample block size larger, but if you make it smaller again, the latency will not go back down: it will stay at the higher value.  What is reported to work, but I haven't tried it yet, is to output sound using more than one Sound object, each with its own block size (and hence its own latency value).  The usefulness of that approach is also not clear.]

Noteflight is a new kind of tool for musicians, composers and educators: an online music notation editor. With Noteflight, anyone can create, share and publish musical scores using nothing more than a web browser. This ease of editing, sharing and publishing are what make Noteflight so different from other notation editors: music notation can finally be used on the Web in as natural and flexible a way as text, images or videos.

Here’s an example of a score that was created in Noteflight, shared, and embedded in this post:

This isn’t just a picture of music, but an interactive music display; for example, you can select and listen to individual notes and measures. Click the logo to see the full-page score, which can be printed. To get a sense of how the score above was created using the Noteflight score editor, please watch this video:

Scores are saved on the Noteflight server, so you and anyone you share them with can access them from anywhere on the Web. The music can be heard as well as seen: Noteflight has its own integrated audio playback, that sounds consistent on every computer. And Noteflight keeps track of all the past versions of your scores, so you can always go back and see where you’ve been.

The Noteflight beta is free and we’re accepting signups. As with all beta software, Noteflight is constantly being improved and extended; this blog is the best place to find out what’s happening (and what’s going to happen next) with Noteflight. Besides filling out the score editing features of Noteflight, we are planning additional services and capabilities for sharing music and organizing content.

There will be more posts coming up here about the development and internals of Noteflight — I’ve had to wait on many of those because the product wasn’t officially launched yet! A few quick techie observations:

• The Noteflight score editor was built with Adobe Flex 3.0.1 and Flash CS3
• The Noteflight server side was built with Ruby on Rails 2, Apache and MySQL and is hosted on Amazon EC2 and S3.
• The Moccasin open-source editing framework (see previous post) is a distillation of many of the music-independent aspects of the Noteflight score editor.
• The StandingWave audio library, previously discussed here (but not yet open-source), is used by the music synthesizer internal to Noteflight.

If the current lack of great overview docs is the bad news, well, the good news about Moccasin is that it’s got a complete sample editor application called Simpleworld as part of the distribution, with both AIR and Flex versions. Both the Moccasin framework and the sample application source code are pretty well commented, also. Here’s the Simpleworld application itself, to give a concrete sense of what Moccasin’s all about — it’s an editor for documents that consist of a bunch of squares of different sizes (I don’t call this world “simple” for nothin’):

Here are some of the things you can do, which show off what Moccasin’s about…

• load and save XML documents on the server. View the one that’s loaded by the above example here.
• Click any square to select it singly, Ctrl/Command-click squares to make discontinuous selections
• Click on the background to add a new square to the world
• Drag any selected square to move all selected squares together
• Drag the mouse across the background to select all squares intersecting that rectangle
• Delete, Cut, Copy or Paste any selected objects
• Resize any individual square by dragging its resize handle (generalizes easily to multiple object handles)
• Undo/redo the last N editing actions
• View the graphical document at any desired magnification using the zoom slider

That’s the externally visible stuff; what about the internals, though? Moccasin is a framework for building applications, not a single application. Some of the above features are built right into Moccasin (like undo/redo or magnification), while others are implemented on top of it (like square objects, or the ability to resize them with a draggable handle).

Moccasin is not a “thin” framework that encapsulates a set of recommended practices like Cairngorm. It’s more of a “thick” framework. Moccasin is aimed at building a particular type of app: one with interactive graphical views that support a lot of mouse gestures, have some notion of compound or nested views, have some notion of a current selection, a clipboard for cut/copy/paste, need robust undo/redo, and must load and save documents to persist the editor state. The views in question are generally not Flex UIComponents in Canvases or Boxes, but lower-level Flash DisplayObjects that can have very arbitrary shapes and layout geometries. Within this domain, Moccasin very much adheres to an MVCS way of thinking: there is a model representing the edited document’s state, a set of views that adjust to the model’s state via events, a controller which changes the model in response to user actions, and a set of services responsible for talking to the world around and outside the application.

Let me devote the rest of this post to a few observations on Moccasin’s treatment of the Model/View relationship. I’ll try to keep up a series of posts over the coming weeks and, with some luck, accumulate some overview materials in fits and starts. Probably the only way it’s going to happen given my schedule these days.

The way Moccasin treats Models and Views is somewhat unique (where by “unique” I mean, “I never tried it before” — I’m sure someone else has!). As usual, one goal here is to absolutely separate view objects from models: views can reference models, but never the other way around. This not only keeps the architecture clean, but it is absolutely essential to permit multiple views of the same model. A second primary goal is that any change to the model should cause the view to adjust itself accordingly without additional effort by the programmer. This involves more than just data bindings: creating a model should cause the view to be added to the display list, and removing a model from the document must cause the view to vanish.

A common approach to model objects in a graphical editor is to make them inherit from some big old AbstractModel superclass with a boatload of common code in it. This code consists of all the gears and wires to dispatch the right events when the model changes, or when its set of children changes, or when some descendant model changes, or to generate a stable ID for use in document persistence… a lot of junk.

By contrast, here is a SimpleWorld-style class (with comments removed for brevity):

    [RemoteClass]
    [Bindable]
    public class Square
    {
        public var x:Number;
        public var y:Number;
        public var size:Number;
    }

Hey, what’s up? This model class has no methods and extends… Object! (Actually, EventDispatcher, due to the Bindable tag, but we’re splitting hairs).

In Moccasin, there are actually two sets of model objects: the domain-specific model (that application programmers work with) and the framework model (that Moccasin deals with). The developer of a Moccasin app only writes code against the domain-specific model, and Moccasin automatically wraps that model object inside another object of type MoccasinModel. MoccasinModel is where the “framework junk” lives, but the application programmer doesn’t see it.

There is also going to be a corresponding View class to draw a Square on the screen — SquareView. Here’s a sense of what a view class looks like:

    public class SquareView extends ShapeView
    {
        public function SquareView(context:ViewContext, model:MoccasinModel=null)
        {
            super(context, model);
            initialize();
        }

        override protected function updateView():void
        {
            super.updateView();

            graphics.beginFill(0);
            var square:Square = model.value as Square;
            graphics.drawRect(0, 0, square.size, square.size);
            graphics.endFill();
        }

        override protected function createFeedbackView():DisplayObject
        {
            return new SquareFeedback(context, model);
        }
    }

Because of the strong decoupling between the view, the framework model and the domain-specific model, application programmers can wind up writing very pure code to manipulate the model that doesn’t have to have any dependencies on Moccasin. You can say something like mySquare.x += 100, and this will cause the framework to adjust the corresponding SquareView object to be redrawn in a different position. Changing an ‘x’ property doesn’t seem very exciting, but consider that it’s not SquareView’s ‘x’ property being changed: it’s the Square’s property. In fact, the relationship between model properties and view properties can be quite arbitrary. We could have some other view of our Squares besides a 2D rendering, perhaps a numerical SquareCoordinatesTable with an X column, and a Y column, and Moccasin would manage that model/view relationship in much the same way.

It’s getting near my lame-old-guy bedtime and I’m flagging. I feel better for having squeezed this post out at the end of a long day though; now to keep it going! Next probable topic: Mediators, Contexts and Controllers in Moccasin.