On Friday, I fly to San Francisco, rent a car, drive to San Rafael, meet spouse who is already there, and then we’ll spend 6 glorious days moseying around the wilds of the Mendocino County and northern Marin coastlines.  Goodbye, work, for a while.  I predict there will be some kind of trip report!

Noteflight is a Flash-based music notation editor, and one of the most often-requested features has been the ability to cut, copy and paste between different documents.  These “clipboard” operations already worked within a single session editing a single document.  However, the clipboard contents did not survive across edits to a series of documents,  nor could they transfer between different documents open in different windows.  In effect, each copy of the application had its own private copy of the clipboard, initialized to emptiness.  Not surprisingly, users don’t like these limitations.

It seemed clear that LSOs would be a good vehicle for this enhancement.  We would write the clipboard to an LSO on every cut or copy, and read it out of the same LSO before every paste. This would not be exactly like a real native system clipboard (since it would be separate from the system clip that contains text, images, etc.) but for the purposes of Noteflight it would seem a lot like one. The challenges we expected to solve (and did) were as follows:  serialization, keeping LSOs small, handling LSO failure gracefully, and application version skew.  There were of course a couple of major surprises, too!

Serialization.  All LSOs are stored in files, and must serialize their data.  They do so using the AMF3 protocol. ActionScript 3 defines how one can manage the serialization of arbitrary Objects using the [RemoteClass] and [Transient] metadata tags (be sure to look those up if you are thinking of serializing objects in a Flex application).  I’ve used them before, but the special problem here was that the clipboard data structures had already been designed and were quite complex, with a number of potential references to objects sitting “out in the application”.  Serializing any of those would result in lots of irrelevant data getting “dragged in” to the LSO and bloating it, so they’d have to be marked as transient.  It would be also necessary to make sure that each and every class requiring a [RemoteClass] tag received it.  Getting all this right in a short amount of time seemed unlikely.

Fortunately I had plenty of existing code to serialize the objects in the clipboard to XML and back, since Noteflight persists musical scores as XML.  So I used Flash’s handy IExternalizable interface to take advantage of this. This is a great technique that allows you to decide how your objects will be serialized. Basically, what I did was kind of like this:

[RemoteClass]
public class AppClipboard implements IExternalizable
{
  private var _data:AppData;  // contents of the clipboard...

  public function writeExternal(output:IDataOutput):void
  {
    output.writeObject(XmlWriter.toXML(_data).toXMLString());
  }

  public function readExternal(input:IDataInput):void
  {
    _data = XMLReader.fromXml(new XML(input.readObject()))
  }
}

(Note the [RemoteClass] tag — you still need it even if you use IExternalizable, or Flash won’t reconsititute the class of your object when you read it back in again.)

With this code written, when you setting one of your shared object’s data properties to an AppClipboard, like this:

  mySO.data.contents = myAppClipboard;

then the writeExternal() method is implicitly used to serialize that object. Same deal goes when reading the shared object back in: readExternal() gets used.

Next challenge:

Keeping the data size low. since there is a default 100K limit on LSO storage. XML has its disadvantages, and verbosity is certainly one of them. I needed to compress my serialized objects somehow. I found a nice way to do that, which works for any serializable objects at all (whether you use IExternalizable or not). Here’s the “write side” of the technique:

  var so:SharedObject = getClipboardSO();
  var bytes:ByteArray = new ByteArray();
  bytes.writeObject(_clipboard);   // write our object to a ByteArray
  bytes.compress();    // compress the ByteArray
  so.data.contents = bytes;   // and write the ByteArray instead
  so.data.version = NotationXmlReader.CURRENT_VERSION;
  so.data.creationTime = _clipboardCreationTime;
  so.flush();

The “read side” works similarly:

  var so:SharedObject = getClipboardSO();
  if (so != null
      && so.data.version == CURRENT_VERSION
      && (_clipboard == null
          || _clipboardCreationTime < so.data.creationTime))
  {
      var bytes:ByteArray = so.data.contents as ByteArray;
      bytes.uncompress();
      _clipboard = bytes.readObject() as AppClipboard;
      _clipboardCreationTime = new Date().getTime();

      // Flush the object after reading it so it doesn't get cached!
      so.flush();
  }

Here you can see the version skew detection, as well as a guard against using a “stale” shared clipboard that is actually older than the local version. That can happen if the “write side” fails to save an LSO due to size limitations, in which case we have to degrade gracefully and use the local clipboard instead of the stale shared one. This means that a single instance will still behave consistently even if it fails to transfer data to other instances.

Surprise #2: note the call to so.flush() on the read side. I did not expect to have to flush() a SharedObject after reading it. However, before putting this flush in, I found that the player would cache the SharedObject and refuse to check to see if it had been updated when reloading it by calling SharedObject.getLocal(). This caused clipboard-style transfer of data between different application windows to fail sporadically. It took a long time to figure this one out.

Apache2. Apache2 prefork is configured to keep a certain number of processes running to respond to HTTP requests that come in. These processes are reused across a number of requests and thus tend to grow over time in their memory consumption to reach the largest amount of RAM demanded by any page served. When using mod_php, the RAM consumption is related to PHP application resources, in this case WordPress. I found that my Apache processes generally were in the 20m range for resident memory size (the RSS column in the ‘top’ display). With the out-of-box Apache2 configuration allowing as many as 15-20 processes, that meant that any large batch of concurrent requests (such as those originating from a search bot) would totally nail my machine. So the most important change I wound up making was to reduce the number of processes:

<IfModule mpm_prefork_module>
    StartServers          3
    MinSpareServers       3
    MaxSpareServers       3
    ServerLimit          10
    MaxClients           10
    MaxRequestsPerChild   1000
</IfModule>

This generally causes me to see around 5-6 processes running, with the possibility of as many as 10.

A number of folks have suggested I run nginx or lighttpd, which I’m sure would be a great idea, but I was strapped for time and in a mood to stick with things that have lots of online recipes and that I already somewhat understood.

Rails. Various approaches are available here as well. At first I was running the mod_passenger module, but I found this gave me a few 30m processes as opposed to the single 50m process I could get by running a single mongrel. So now I’m running a single mongrel and it’s working fine.

MySQL. Haven’t tuned this much yet, but I’m not seeing any problems.

WordPress. Caching is enabled on the advice of a number of people that this will decrease MySQL resource consumption.

Fortunately, we now have the prospect of an American President who can speak in intelligible sentences… which means that you can turn all of your positive energy to getting Adobe to fix this bug. Please vote on it at:

https://bugs.adobe.com/jira/browse/FP-985

Future generations thank you! (Future generations of what, you may ask?)

The merge went smoothly and the 3.2.0 compiler seems to work like a champ — well, at least, it does everything that the 3.0.x version did. What’s more, a sneak peek at Flex Builder 3.2 revealed that they’ve fixed an irksome bug in the Flex Compiler project options dialog, and it’s now possible to enable and disable coverage instrumentation using the compiler options field.

That’s just the part I did. Alex has been busy working on some very nice changes to the CoverageViewer, which should greatly reduce the performance problems people were seeing during data collection before. We’ve also made it much easier to merge reports from multiple test runs. A few more fixes are needed but things are looking up…

My previous hosting company — referred to obliquely in the title of this post — blew up in this past week, to the point where I had to make an abrupt decision to pull everything off their servers. Here’s more or less what happens, starting Friday:

• With no warning, my Virtual Private Server account is moved to a new domain name and static IP. (You might wonder what the meaning of “static IP” is under these circumstances.)
• The old VPS instance continues running and is still accessible at its name and IP, so I assume everything is fine. However, all activity on that instance (including SVN commits) are written to a separate disk not associated with the new VPS.
• Those of my sites which are running on the new VPS instance break, because the new instance is misconfigured and can’t reach the DB server. Also its postfix install turns out to be broken due to bad permissions setup. So the new VPS is basically hosed.
• I wait over 48 hours for support to return my emails while my sites are down. I do not even receive an auto-reply for almost 36 hours.
• The same backup cron job is running on both VPS instances, unbeknownst to me. The job on the new instance backs up stale data that overwrites on the backups from the old VPS (which is still in use, because I haven’t been told yet which is the “real” instance).
• Support fixes the new VPS so it can reach the DB server, and the old VPS is removed. I switch all remaining services to the new VPS but start the process of pulling out.
• Support tells me that I was mistakenly charged for a hosting plan along with the VPS account, cancels the hosting plan, and credits my account with the prepaid balance. With the disappearance of the hosting plan, all my databases and user accounts promptly disappear before I can back them all up and migrate them elsewhere. All sites down again.
• Support reinstates the hosting plan. I pull everything remaining over to Slicehost and get it running again there.

It’s sad, because this hosting provider used to be a good deal, and delivered exactly what I needed at the point when I first started using them. I wish I could say the same today.

Go see David.  Go see David.  Go see David.

Is there anything about “Go see David” that you don’t understand?  He’s a dynamic, intelligent and very, very informative speaker.  Hope to see you there!

Why use a strongly typed proxy? A common use of such a proxy is to create a “smart wrapper” around some underlying object with a known interface that looks and acts (and code-hints) exactly like the underlying thingamabob, but does some other stuff around every method call or property access. You might want a smart wrapper that logs or records all the function calls (as well as executing them). You might want a wrapper that does a try/catch around every call. You might want a wrapper that serializes the call over a stream to some remote destination.

(If this sounds like what some call “aspect-oriented programming”, well, it is similar to one aspect of aspect oriented programming :-)

Anyway, we can’t use AS3’s Proxy to solve this problem, so what to do? We’ll start with a simple approach, but make it more general. If our interface looked like this:

public interface ISmashable
{
    function smash(force:Number):String;
}

and we wanted to create a smart wrapper that logged all function calls around any ISmashable implementation, we might create something like this:

public class SmashableLoggingProxy implements ISmashable
{
    private var _smashable:ISmashable;
    public function SmashableProxy(smashable:ISmashable)
    {
        _smashable = smashable;
    }

    public function smash(force:Number):String
    {
        trace("smash():", force);
        var result:String = _smashable.smash(force);
        trace("    result:",result);
        return result;
    }
}

Then, if we did something like var proxy:ISmashable = new SmashableProxy(realSmashable); we could treat the resulting proxy as an ISmashable everywhere in our application, Every time we called, say smash(5) method on proxy, not only would realSmashable.smash(5) be called but we’d see some nice trace output as well.
If you have the time to create a lot of wrapper methods like this (or write a fancy code generator to save the trouble), that’s fine.  Another way to go, however would be to create a “generic” smart wrapper base class:

public class SmartProxy
{
    private var _obj:Object;
    private var _functionNames:Dictionary;

    public function SmartProxy(obj:Object)
    {
        _obj = obj;

        // set up Dictionary that goes from functions in this wrapper to function names
        _functionNames = new Dictionary();
        var typeInfo:XML = describeType(this);   // returns metadata for subclass interface
        for each (var functionInfo:XML in typeInfo.method)
        {
            var name:String = functionInfo.@name;
            _functionNames[this[name]] = name;
        }
    }

    protected function handleInvocation(args:Object):*
    {
        // 'callee' is a special AS3-defined property of an 'arguments' object
        // that provides the Function which was called
        var functionName:String = _functionNames[args.callee];
        return applyFunction(functionName, args);
    }

    protected function applyFunction(functionName:String, args:Object):*
    {
        // This looks up the name of the function that was called, uses it to
        // obtain the Function in the underlying object as a property, and calls it.
        return (_obj[functionName] as Function).apply(_smashable, args);
    }
}

Now extend this to implement our interface:

public class SmashableProxy extends SmartProxy implements ISmashable
{
    public function SmashableProxy(smashable:ISmashable)
    {
        super(smashable);  // sounds like a Nintendo game, eh?
    }
    public function smash(force:Number):String
    {
        return handleInvocation(arguments);  // arguments is a special AS3 language keyword!
    }

    // (if our interface had other functions, they would all have the same body!)
}

The above SmashableProxy acts exactly like the underlying object. What’s the point? Well, given a base class of this kind, one can create smart wrappers of any variety with very little additional effort:

public class SmashableLoggingProxy extends SmashableProxy
{
    public function SmashableProxy(smashable:ISmashable)
    {
        super(smashable);
    }

    override protected function applyFunction(functionName:String, args:Object):*
    {
        trace(functionName, "():", args);
        var result:String = super.applyFunction(functionName, args);
        trace("result:",result);
        return result;
    }
}

The astute reader may notice that logging is generic, and wonder if it should be handled in the SmartProxy, not in this subclass.  So a further extension of this idea might  introduce a separate invocation handler that can supply generic handling for all method invocations,  and is a property of the SmartProxy.  Java in fact has exactly such a class for its Proxies.

In another post soon, I’ll describe how this fits into the testing strategy for Noteflight, which has a large automated test suite of recorded UI actions. We use this suite to validate the program before every checkin — it is a very complex editor, and it’s all too easy to fix one thing and break another, so the test suite makes sure we don’t do that!