Java Script xUnit style frameworks comparison

In this installment of the series I’ll look at xUnit frameworks. Comparison of BDD frameworks and standalone mock frameworks will follow in subsequent posts.

JsTestDriver
http://code.google.com/p/js-test-driver

This one is a very strange beast. Its main focus, as claimed by authors, is command line and continuous integration – you can think of jsTestDriver more as a robust test runner platform rather than a typical assertion framework. This is very visible in its design – its client-server architecture based runner is as robust as it may get, definitely one of (or the) best on the market, while assertion framewrok, although solid, is relatively bare bone.

SYNTAX

TestCase("MyTestCase", {
    setUp: function() {
        //...
    },

    testAssertionWithMessage: function() {
        assertTrue("this is not true", false);
    },

    testAssertionWithoutMessage: function() {
        var actual = 5;
        assertEqual(5, actual);
    },

    tearDown: function() {
        //...
    }
};

PROS

  • Robust Continuous Integration support: parallel execution in many browsers at once (both local and remote) with aggregated result reporting; designed with speed in mind (keeps everything permanently loaded in browser, reloads only files that changed – what gives a huge speed boost by avoiding frequent re-parsing of code)
  • Allows replacing its assertion layer with another framework. This way jsTestDriver acts only as a test runner while actual test suites are written using alternative syntax. This offers possibility to replace not so pretty jsTestDriver syntax with something more fancy (even changing its flavor from xUnit to BDD in case of integrating with Jasmine) and also makes porting your existing code to jsTestDriver a breeze if you are lucky to use one of supported frameworks. Currently though, only a few integrations are supported (as of time of this writing only QUnit, YUI Test and Jasmine).
  • Integrates with Eclipse and IntelliJ IDEA. If you are using one of these, it may compensate for the lack of HTML runner (bad luck for NetBeans users though).
  • Automatically resets DOM between tests. Also provides some support for HTML fixture loading (although rather very basic, based on comments: if you have HTML inside a comment block in your test, it will get automatically loaded into DOM).
  • No need to create any HTML wrappers to run tests.
  • There is a code coverage plugin for jsTestDriver available (http://code.google.com/p/js-test-driver/wiki/CodeCoverage).

CONS

  • No HTML runner. If you happen to use IDE that is not supported by jsTestDriver, you must read test results from the console, with its ugly and hard to read formatting of error messages.
  • A bit over-complicated to run on a developer machine. For CI this is no problem, but having to keep open two consoles (one to control a server, second to display test results) plus one or more browsers on a local workstation is a little overkill for me.

CONCLUSION

This is a really solid framework, with simple, but nice and clean syntax and a couple of unique features. And although its CI oriented client-server architecture may feel too heavy for some usage scenarios, it is a real deal for bigger, more complex projects. Among xUnit style frameworks this is definitely one of the best choices.

QUnit
http://docs.jquery.com/QUnit

This is an “unofficial official” testing framework for jQuery. It’s mature and quite established on the market (also partly due to the popularity of jQuery). It is rather simplistic, containing by design only most core set of features (with two notable exceptions: support for HTML fixtures and testing of asynchronous calls).

SYNTAX

test("a basic test without module", function() {
    var value = "hello";
    equals("hello", value, "We expect value to be hello");
});

module("Some Module", {
    setup: function() {
        //...
    },
    teardown: function() {
        //...
    }
});

test("first test within module", function() {
    ok(true, "all pass");
});

test("second test within module", function() {
    ok( true, "all pass" );
});

PROS

  • HTML runner with nice output formatting.
  • Automatic clean-up of DOM between tests. Some support for HTML fixtures – content of predefined DIV, if present inside runner’s page HTML, is re-created between tests (although this is not fully clean when you run multiple suites at once – all fixtures for all suites must be cluttered in HTML in such case, instead of being stored separately and loaded on demand).
  • Support for testing asynchronous code.
  • Integrates with jsTestDriver.

CONS

  • Non standard (and non intuitive) syntax. Test methods are placed outside, instead of inside of module. Assertion have very custom naming (e.g. “ok” instead of “assertTrue”). Surprising “expect” method (setting expectations at the beginning of a test on how many calls to other assertions will be made in this test).
  • Absolutely barebone set of assertions (only 3 assertions available: “ok”, “equals” and “same”).
  • No command line runner (and no support for CI).
  • Tests pollute global namespace.

CONCLUSION

Being very simple and being a standard platform for jQuery core tests may seem tempting at first, but this framework is so simplistic, and has such a strange, custom syntax, that I’ll not recommend it. Built-in support for HTML fixtures is a plus, but the way it is implemented makes it useful only for a small test suites, so this doesn’t change the final verdict.

jqUnit
http://code.google.com/p/jqunit

jqUnit is just a wrapper around QUnit framework. It doesn’t add any new features, only wraps the QUnit with a different, JSUnit compatible syntax.

PROS

  • The same as for QUnit (although I’m not sure about jsTestDriver integration).
  • Fixes a bit “weird” QUnit syntax. Allows easy porting of your old JsUnit code.

CONS

  • The same as for QUnit.

CONCLUSION

If you really need to port your huge code base from JsUnit, this may be an interesting choice. Otherwise it is not recommended.

YUI Test
http://developer.yahoo.com/yui/yuitest

YUI Test is a part of Yahoo YUI library. It is mature and very actively maintained (as well as YUI library itself). Although it uses YUI library namespace, it can be used also standalone.

SYNTAX

var oTestCase = new YAHOO.tool.TestCase({
    name: "TestCase Name",

    // Special instructions
    _should: {
        ignore: {
            testSomethingUnimportant: true //ignore this test
        },

        error: {
            testSomethingExceptional: true //this should throw an error
        }
    },

    setUp : function () {
        //...
    },

    tearDown : function () {
        //...
    },

    testSomething: function () {
        YAHOO.util.Assert.areEqual("Something", this.someValue, "Value should be 'Something'");
    },

    testSomethingUnimportant: function () {
        //this test is ignored
    },

    testSomethingExceptional: function () {
        methodThatThrowsException();
    }
});

PROS

  • Robust set of assertions.
  • Built-in mocking of mouse and keyboard events.
  • Supports asynchronous tests.
  • Not only test cases but also test suites can have setUp and tearDown methods – this allows almost BDD like nesting (however with not so nice syntax as in case of BDD frameworks).
  • Integrates with jsTestDriver.

CONS

  • Very ugly HTML runner. It tries to emulate the look of a console – with all its readability problems. (However, YUI Test runner provides very robust support for test result serialization, with option to pass serialized results to a given URL, so it should be possible to create your own pretty printer).
  • No command line runner. (Although, again – robust support for test result serialization should allow you to code your own support for CI / command line invocation).
  • YUI related namespacing. Having to write “YAHOO.util.Assert.areEqual” instead of just “assertEqual” is awkward. It may be mitigated a little though, by doing something like:
    var Assert = YAHOO.util.Assert;
    Assert.areEqual(...);
    
  • Strange, non standard “_should” syntax for “special” instructions (like ignoring tests or expecting exceptions to be thrown).
  • No support for HTML fixtures.

CONCLUSION

YUI Test is a very solid framework. If the lack of robust Continuous Integration support is not a show stopper for you, it is definitely worth considering. A couple of minor syntax annoyances are a small crack on the surface of otherwise polished product, however they are far from being serious enough to discourage you from using it.

JsUnit
http://www.jsunit.net

One of the first, most “traditional” xUnit type frameworks. It is still actively maintained, but it’s starting to show its age.

SYNTAX

function setUp() {
    //...
}

function testAddition() {
    assertEquals("2 plus 3 is 5", 5, add(2, 3));
}

function tearDown() {
    //...
}

PROS

  • Based on JUnit for Java, so it has “standard” syntax.
  • HTML runner.
  • There is some support for server-based testing in several browsers at once (via JUnit or Ant).

CONS

  • Tests methods not encapsulated in classes (pollute global namespace).
  • Very simplistic regarding features (no fixtures, no async testing etc.)

CONCLUSION

Nothing interesting here. This framework might be popular at a time, but now there are other, younger and much better contenders. Not recommended.

Other frameworks

  • jsUnity (http://jsunity.com)
    Seems quite promising, but for now it is a bit too young (as the time of this writing current version is 0.6). It has a clean, readable syntax with a nice set of assertions, but apart from this it is still in a very basic shape: no advanced runners, no CI integration etc. – just a minimal set of  functionality you can expect from a testing framework. On the one hand, there were several updates to this framework, on the other, the newest one is about a year old – so it is difficult to tell if this project is still actively maintained. If so, it may be worth looking at in future.
  • JsUnit (berlios) (http://jsunit.berlios.de)
    Not updated for ages… Probably no more actively maintained.
  • JsUnitTest (http://jsunittest.com)
    Never reached full maturity (latest version is 0.7.3, documentation is sparse) while at the same time not updated for a long time – seems an abandoned project.
  • J3Unit (http://j3unit.sourceforge.net)
    Declared to be in beta stage, but last update was 4 years ago… Seems abandoned.

The Final Verdict

There are two xUnit frameworks for Java Script that stand out from the crowd: jsTestDriver and YUI Test. YUI Test has a more feature packed assertion framework (with user input events mocking, async tests etc.) while jsTestDriver distinguishes itself with its awesome support for multi-browser server-based runner.

The final verdict?

JsTestDriver.

While YUI Test features may seem attractive, most of them is not so hard to mimic by hand, especially when using jQuery or a similar JS framework (e.g. simulating mouse and keyboard events in jQuery is dumb simple). On the other hand, hand-coding jsTestDriver’s support for distributed, parallel testing is a completely different pair of boots. And it’s a must have on any reasonable complex project, especially when using CI.

QUnit also stands out from the crowd in one aspect – its very neat and readable HTML test runner, which may be quite tempting. However, lack in other areas make it suitable only for small, simple projects.

It is worth noting that both QUnit and YUI Test integrations are supported by jsTestDriver – so you can have best of both words if you want – e.g. jsTestDriver’s runner with YUI Test’s robust set of assertions. However, none of xUnit frameworks provides enough advantages over standard jsTestDriver’s assertion framework to justify additional hassle – so, my final verdict is a plain jsTestDriver (at least as far as xUnit style frameworks go – BDD style frameworks are a completely different story).

Also, for a complete solution you need a mocking framework (not built-in any of xUnit frameworks).

Standalone mocking frameworks, as well as BDD flavor frameworks, will be compared in detail in next installments of this series.

6 Comments

  1. […] http://testdrivenwebsites.com/2010/04/19/java-script-xunit-style-frameworks-comparison/ […]

  2. […] der Webseite Test Driven Websites bin ich auf einen schönen Javascript Unit Testing Framework vergleich gestossen. Der Favorit war […]

  3. […] bei den Testing Frameworks, gibt es auch einige Mocking Frameworks. Ich habe mich wieder über TestDrivenWebsites inspirieren lassen und mich für JsMockito entschieden, welcher ein Klon vom entsprechenden JMock […]

  4. You may be interested to know that YUI Test now has tools for both code coverage and CI:

    http://www.yuiblog.com/blog/2010/11/09/introducing-the-new-yui-test/

  5. Chutipunt said

    I wasn\’t aware of the pick-and-choose functionality of ExtJS, so tnakhs for pointing that out! I\’ve been experimenting with jQuery lately, and it also has something similar where it will build you a customized ZIP file.I\’m really looking forward to Ext 3.0. Until then, I\’ll see you on the Ext boards

  6. […] http://testdrivenwebsites.com/2010/04/19/java-script-xunit-style-frameworks-comparison/ […]

RSS feed for comments on this post

Comments are closed.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: